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Abstract 

Interactive encoding and decoding based on binary low-density parity-check codes with syndrome 
accumulation (SA-LDPC-IED) is proposed and investigated. Assume that the source alphabet is GF(2), 
and the side information alphabet is finite. It is first demonstrated how to convert any classical universal 
lossless code C n (with block length n and side information available to both the encoder and decoder) 
into a universal SA-LDPC-IED scheme. It is then shown that with the word error probability approaching 

sub-exponentially with n, the compression rate (including both the forward and backward rates) of 
the resulting SA-LDPC-IED scheme is upper bounded by a functional of that of C n , which in turn 
approaches the compression rate of C n for each and every individual sequence pair (x n ,y n ) and the 
conditional entropy rate H(X|Y) for any stationary, ergodic source and side information (X, Y) as the 
average variable node degree fof the underlying LDPC code increases without bound. When applied to the 
class of binary source and side information (X, Y) correlated through a binary symmetrical channel with 
cross-over probability unknown to both the encoder and decoder, the resulting SA-LDPC-IED scheme 
can be further simplified, yielding even improved rate performance versus the bit error probability when 

1 is not large. Simulation results (coupled with linear time belief propagation decoding) on binary source- 
side information pairs confirm the theoretic analysis, and further show that the SA-LDPC-IED scheme 
consistently outperforms the Slepian-Wolf coding scheme based on the same underlying LDPC code. As 
a by-product, probability bounds involving LDPC established in the course are also interesting on their 
own and expected to have implications on the performance of LDPC for channel coding as well. 

Index Terms 

Belief propagation decoding, distributed source coding, entropy, interactive encoding and decoding, 
low-density parity-check code, rateless Slepian-Wolf coding, syndrome accumulation. 



January 26, 2012 



DRAFT 



3 



I. Introduction 

Recently, the concept of interactive encoding and decoding (IED) was formalized in jl], @- When 
applied to (near) lossless one way learning (i.e. lossless source coding) with decoder only side information, 
IED can be easily explained via Figure [T] where X denotes a finite alphabet source to be learned at the 
decoder, Y denotes another finite alphabet source that is correlated with X and only available to the 
decoder as side information, and R denotes the average number of bits per symbol exchanged between 
the encoder and the decoder measuring the rate performance of the IED scheme used. As evident from 
Figure [T] IED distinguishes itself from non-interactive Slepian-Wolf coding (SWC) in the fact that two- 
way communication is allowed in IED. 
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Fig. 1. Interactive encoding and decoding for one way learning with side information at the decoder 



By allowing interactions between the encoder and the decoder, IED has several advantages over SWC 
|TJ, Q. For example, in comparison with SWC, it was shown jl], (2j that IED not only delivers better 
first-order (asymptotic) performance for general stationary, non-ergodic source-side information pairs, but 
also achieves better second-order performance for memoryless pairs with known statistics. Furthermore, in 
contrast to the well known fact that universal SWC does not exist, it was shown [2] that coupled with any 
classical universal lossless code C n (with block length n and with the side information available to both 
the encoder and decoder) such as the one in (3), one can build an IED scheme which is asymptotically 
optimal with respect to the class of all stationary, ergodic sources-side information pairs. Indeed, the 
corresponding IED scheme achieves essentially the same rate performance as that of C n for each and 
every individual sequence pair (x 11 , y n ), even though the side information is not available to the encoder 
in the case of IED, while the word decoding error probability can be made arbitrarily small. 

The above advantages make IED much more appealing than Slepian-Wolf coding to applications where 
the one-way learning model depicted in Figure [T] fits. However, the IED schemes constructed in (T|, @ 
do not have an intrinsic structure that is amenable to implement in practice. A big challenge is then 
how to design universal IED schemes with both low encoding and decoding complexity. To address this 
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challenge partially, linear IED schemes, which use linear codes for encoding, were later considered in 
Q. The encoder of a linear IED scheme can be conveniently described by a parity-check matrix. Based 
on different random matrix ensembles, two universal linear IED schemes were proposed therein. The first 
universal linear IED scheme proposed in [4] makes use of Gallager-type of matrix ensembles, where each 
matrix element is generated independently, selects randomly a matrix from such an ensemble, and then 
divides the selected matrix into several sub-matrices, each of which is used to generate new syndromes 
in each round of interaction. In the second universal linear IED scheme proposed in [4], Gallager- 
type ensembles are extended into vector-type ensembles, where each column of matrices is generated 
independently, and a matrix is generated in such way that each of its sub-matrices is randomly picked 
from such a vector-type ensemble; in each round of interaction, new syndromes are then generated by 
applying syndrome accumulation (described in Q) once to each and every of those sub-matrices. Define 
the density of a linear IED scheme as the percentage of non-zero entries in its parity-check matrix. It 
was then shown j4j that there is no performance loss by restricting IED to linear IED and even to linear 
IED with density fi(S^), where n is the block length. Thus the encoding complexity of universal IED 
can be kept as low as O(nlnn). 

Although linear IED considered in Q tackles its encoding complexity very well, its decoding com- 
plexity is largely untouched due to the adoption of maximum likelihood (ML) decoding, which results 
in exponential decoding complexity with respect to block length n. One of the main purposes of this 
paper is to address the issue of decoding complexity by building IED schemes from linear codes with low 
decoding complexity. This leads us to consider low-density parity-check (LDPC) codes, due to their linear 
complexity decoding based on belief propagation (BP) and successful application to fix-rate Slepian-Wolf 
coding (5) @ Q (8J. 

An LDPC code is a linear code with a sparse parity check matrix, each of whose rows and columns 
has only a finite number of non-zero elements with respect to its block length. Important parameters of 
an LDPC code include the ratio between the numbers of rows and columns (called Slepian-Wolf rate), 
and the portions of rows and columns with certain number of non-zero elements (called the check and 
variable degree distributions of the LDPC code). Given a block length n and a Slepian-Wolf rate, one 
way to generate an LDPC code with the given Slepian-Wolf rate, is to randomly select a matrix as its 
parity check matrix from an ensemble in which all matrices share the same Slepian-Wolf rate, and check 
and variable degree distributions. 

Since rows and columns of parity check matrix of an LDPC code are not generated independently, 
the approach of dividing the whole matrix into several sub-matrices adopted in [4] can not deliver good 
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results from both theoretical and practical perspectives. To overcome this problem, in this paper, we 
shall modify syndrome accumulation (SA) used in |4} to adapt the encoding rates of the LDPC code 
for IED. The resulting scheme is called an interactive encoding and decoding scheme based on a binary 
LDPC code with syndrome accumulation (SA-LDPC-IED); its performance is then analyzed theoretically 
and evaluated practically based on minimum coding length decoding and BP decoding, respectively. It 
is shown that coupled with any classical lossless code C n (with side information available to both the 
encoder and decoder), one can always construct an SA-LDPC-IED scheme such that 

• the word decoding error probability approaches sub-exponentially with n; and 

• the total rate (including both the forward and backward rates) of the resulting SA-LDPC-IED scheme 
is upper bounded by a functional of that of C n , which in turn approaches the compression rate of 
C n for each and every individual sequence pair (x n ,y n ) and the conditional entropy rate H(X|Y) 
for any stationary, ergodic source and side information (X, Y) as the average variable node degree 
I of the underlying LDPC code increases without bound. 

When applied to the class of binary source and side information (X, Y) correlated through a binary 
symmetrical channel with cross-over probability unknown to both the encoder and decoder, the resulting 
SA-LDPC-IED scheme can be further simplified, yielding even improved rate performance versus the bit 
error probability when I is not large. 

It should be pointed out that in the literature (see for example Q, pUJ , [11 1, and references therein), 
there have been several attempts towards building rateless (or rate- adaptive) SWC schemes using LDPC 
codes. Specifically, the technique of SA was used to construct the so-called LDPC A codes in [11]. Our 
SA-LDPC-IED schemes differ from the rateless SWC schemes in the following aspects: 

• We are concerned with the total rate defined as the number of bits exchanged between the encoder 
and the decoder per symbol, while only the forward rate (from the encoder to the decoder) is 
considered in rateless SWC schemes. 

• We assume that the joint statistics of source and side information are unknown to both the encoder 
and decoder, while the joint statistics are available for decoding in rateless SWC schemes. 

• We provide theoretical analysis for our SA-LDPC-IED schemes, while the performance of those 
rateless SWC schemes has been evaluated mainly through simulation. 

The rest of the paper is organized as follows. In section [n| several definitions and convention are 
introduced to facilitate the following discussion. The concept of syndrome accumulation is revised and 



SA-LDPC-IED schemes are constructed in section III while the performance analysis is performed in 
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section IV in terms of the forward and backward rates versus the word error probability for individual 
sequence pairs (x n ,y n ) and stationary, ergodic source-side information pairs (X,Y), and in section 
|V| in terms of the forward and backward rates versus the bit error probability for binary source-side 



information pairs (X, Y) correlated through a binary symmetrical channel. The section VI is devoted to 



practical implementation and simulation results, followed by the conclusion in section VII 



II. Preliminaries and Convention 

In this section, we first set out our notation for the paper and then review some concepts related to 
LDPC codes. 

Throughout the paper, we use uppercase and lowercase letters to denote random variables and their 
realizations, respectively. Let B be the binary alphabet, and B + the set of all finite strings from B. Let 
B n denote the set of all strings of length n from B. Similar notation applies to other alphabets (e.g. 
y) as well. A vector of dimension n is represented by a letter with superscript n, e.g. b n ; a matrix of 
dimension mxnis represented by a bold letter with subscript mxn, e.g. H mxn . Whenever superscripts 
and subscripts are clear from context, they will be omitted. For example, when there is no ambiguity, 
we shall simply write b n as b and H mx „ as H. The entropy function based on logarithm with bases 2 
and e will be denoted by H(-) and H c (-), respectively. We will denote by E(-) the expectation operator, 
and by wt(-) the Hamming weight function counting the number of non-zero elements in a vector. 

For any two sequences {a,}™ =1 and {b{}f =1 , we write a n ~ b n if 

lim — — = 1 . 

n— >oo b n 

Furthermore, for any positive integer x, define 

a if x is even 
vr(x) = { (2.1) 
1 otherwise. 



Consider now a linear block code with its parity check matrix H mx „. The tanner graph |T2| of the code 
(or equivalently, its parity check matrix) is a bipartite graph consisting of two sets of nodes {i>j}™ =1 and 
{cj}™ =l , namely, variable and check nodes, where for any i and j such that 1 < i < n and 1 < j < m, Vi 
and Cj, representing the i-th column and j-th row of H TOXn respectively, are connected if and only if the 
element hj% of H. mxn located at z-th column and j-th row is equal to 1. Note that the degree of a node 
in a graph is the number of edges connected to it. Let {k : I < i < L} ({rj : 1 < j < R}, respectively) 
be the set of degrees of all variable nodes (check nodes, respectively) in the tanner graph of H mxn . 
Furthermore, let Aj (Pj, respectively) denote the number of variable nodes (check nodes, respectively) 
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with degree Zj ( Tj, respectively) in the tanner graph of H mxn . Then we call ({Aj}, {k}) (({Pj}, 
respectively) the variable (check, respectively) degree distribution from a node perspective of H mxn (and 



its tanner graph) [13]. Define polynomials A(z) and P(z) as 

L 



i=i 

and 

R 

The tanner graph is said to be sparse and accordingly its corresponding code is said to be a low-density 
parity-check code if A'(l) is in the order of 0(n), where A'(l) = J2i=i Ai4 is the total number of edges 
in the tanner graph. Normalizing {Aj} and {Pj} by the total numbers of variable nodes and check nodes 
respectively, we get normalized variable and check degree distributions L(z) and R(z): 

ji A(z) 



L(*) = J> 



z 



A(l) 
and 

j 

where Li and Ri represent the percentages of variable and check nodes with degrees k and ri respectively. 

Given m, n, and (normalized) variable and check degree distributions L(z) and R(z) satisfying 
nL'(l) = mR'(l), let T-L m ,n,L{z),R(z) (simply / H n ,L(z),R(z) if m = n) denote the collection of all m x n 
parity check matrices with normalized variable and check degree distributions L(z) and R(z). Without 
loss of generality, we only consider those matrices such that the degrees of rows and columns do not 
decrease with their indices. (In other words, i > j implies the degree of the z-th row (or column) is not 
less than that of the j-th row (or column).) Then an LDPC code of designed rate 1 — m/n is said to 
be randomly generated from the ensemble with degree distributions L(z) and R(z) if its parity check 
matrix H mxn is uniformly picked from T-L m ,n,L{z),R(z)- m this paper, we consider only such generated 
LDPC codes. 

The performance of an LDPC code (under ML and BP decoding) depends largely on degree distribu- 
tions of the ensemble it is picked from. According to the analysis in p3| , a class of degree distributions, 
called check-concentrated degree distributions, are of special interest due to their superior performance, 
where given a variable node degree distribution, the check node degree distribution is made as concentrated 
as possible. In this case of l~L n Mz),R(z)' given L(z), R{z) is determined as follows: 

R{z) = Riz ri + R 2 z r2 



January 26, 2012 



DRAFT 



8 



where 



n = [i\ 
T2 = \i] 

Ri = i + lfj - r 

R 2 = T- [l\ 

L 

T=L\l) = Y,Lik ■ 

i=i 

Under this circumstance, r H n ,L{z) > R(z) is simply referred as to 7i n u z ). 



and 



III. Interactive Encoding and Decoding based on Syndrome Accumulation 
A. Syndrome Accumulation 

The concept of syndrome accumulation has been introduced in [4]. To clarify our following discussion, 
we revise this concept here. 

Suppose a syndrome vector s n = H nxn x n is given, where s 11 consists of n syndromes s\S2 ■ ■ ■ s n , 
and H nxn is an n x n matrix. To facilitate the discussion below, we assume that n is a power of 2, i.e. 
2 T for some positive integer T. Let M = {1, 2, . . . , n} and V = {Ai, A2, . . . , Ai-pi} where V forms a 
partition on M with each Aj as a subset of AT and \V\ as the number of elements in V. A4 is also called 
a cell in V, and we use |Aj| to represent the cardinality of Aj, i.e. the number of indices in Aj. Now 
given s n and V, we can form a new syndrome vector s^l, which is called an accumulated syndrome 
vector, in the following way: 

( ft \ 



So 



\ ~ S W\ J 

Sj for 1 < % < \V\ 
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The derivation below shows that s\ v \ is indeed a syndrome vector: 

h 



S W\ = 



ieA 2 s j 



( EjeAi ELi h 3kXk ^ 
EjeA 2 Efc=i ^-jfc^fc 

\ EjeA|p, Efe=i ^jfc^fc / 

( TJLiT.j^hjkXk \ 

Efc=i EjeA 2 hjkXk 

\ Efc=i Ej6A|„i ^jfc^fc / 



u'eA, 



l<»<|7'|,l<fc<n| 



^2 



where /i^ is the element in the j'-th row and fc-th column of H nxn , and Xk is the fe-th element in x n . 
Also, Hp defined above is the parity check matrix corresponding to the partition V. 

To proceed, we introduce a sequence of partitions V\Pi ■ ■ ■ V n . (Later on, it can be seen that this 
sequence effectively represents the procedure of encoding of SA-LDPC-IED schemes.) The sequence 
V1P2 ■ ■ ■ V n is generated in a recursive manner, depicted below: 

• V x = {AA}. 

• Suppose Vi = {Aj i, Aj 2, • • • , Aj j} has been generated. Let ji = 2(i - 2L lo §^J) + i. Split A i:ji 
equally into two parts, Ajj. + and Ajj 4 _, where Ajj.+ (Ajj._) consists of the first (second) half of 
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elements in Ajj., ordered by their values. 
• Vi+i = {Ai+i,i, K+1,2, • • • , A i+ i >i+ i} is generated as below: 

- A i+ i jfc = k i>k for 1 < k < ji. 

- K+iji = 

- A i+ ij 4+ i = Ajj._. 

- Aj + i )fc = A ijfc _i for ji + 1 < k < i + 1. 

Note that since we assume n = 2 T for some integer T, | A^j is also a power of 2 for 1 < i < n, 1 < k < i. 
Moreover, for 1 < i < n, \A i>kl \ = 2|Aj jfc2 | = 2 T ~Li°g2*J always holds for ji < ki < i and 1 < k 2 < ji— 1. 
Therefore, the splitting of Ajj. can always be applied. In fact, 

A i>fc = { (k - 1)2 T ~ ^ *1 + 1, . . . , A;2 T ~ Tiog 2 il | 

for 1 < k < ji, and 

A llfc = {(ji - 1)2 T -^^1 + (fe - j t )2 T -^^ + 1, 
. . . , (ji - 1)2 T " ^ ^ + (k- ji + l)2 T ~ ^ J J } 

for ji < k < i. 

Now given s n = H nxn x n and V\ V2 • • -V n , we can generate a sequence of accumulated syndrome 
vectors s\s\ . . . s™, where the upper scripts represent the dimension and lower scripts indicate which 
partitions the syndromes are associated with. The upper scripts, which always equal to the lower scripts, 
are dropped for simplicity. Now for any Sj, we use Sij to represent its j-th element. In fact, this procedure 
can be done recursively as above, where 

Si = h,i = Yl 8 i 

and is generated by replacing sij i with Si+i,^ and Sj+i^+i. Moreover, since {Aj + ij., Aj + ij. + i} 
is a partition on Ajj., we have 

= Sj+lj'i + Sj+lji+l 

and therefore, if Sj is known, only one of Sj+i Jt and Sj+ij.+i is needed to calculate Sj+i. We call 
Si+ijt as the augmenting syndrome from §i to denoted by a^+i. We also adopt the convention 

that ai = si ; i for convenience. In addition, according to the discussion above, §i = Hp z x n , where H-p. 

(i) 

can be determined by H nxn and T>i. For clarification, we refer to H-p. as H^ n , where the lower script 
indicates its dimension. 
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Fig. 2. Binary Tree Structure of Syndrome Accumulation 
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By remark 7 in |4], a binary tree can be associated with V1V2 • • • V n or S1S2 • • ■ s n , shown in figure [2] 
where each node represents a subset of M. Let v and A(u) be a node and its associated set. {A(uj), A(u r )} 
forms a partition of A(v) when and v r are the left and right child nodes of v. Moreover, let v(A) be 



the node associated with the set A, and d v be the depth of a node v. Then |A| = 2 



(A) 



B. Interactive Encoding and Decoding Schemes 

In light of LDPC codes, we consider only binary sources. That is, the source alphabet X is binary. 
However, the side information alphabet y could be arbitrary. For any x n E X n , let x n be the complement 
sequence of x n , i.e., the sequence having hamming distance n from x n . Let H nxn be the parity check 
matrix of a LDPC code randomly generated from the ensemble % n u z ) for some L{z). Let H' nxn 
and H" nH( ^ +A ^ xn be matrices from Gallager parity check ensemble (the set of matrices with each 
element generated independently and uniformly from £>), where 0<r? n <l, 0<e< 0.5, and nH(e) is 
assumed to be an integer. Furthermore, let V\Pi ■ ■ ■ V n be the partition sequence described in the previous 
subsection. Based on the concepts introduced above, we are now ready to describe our SA-LDPC-IED 
scheme X n , which is presented in details in Algorithm [T] below, where x n is the source sequence to be 
encoded, y n G y n is the side information sequence available only to the decoder, and A is an integer to 
be specified later such that ^ lS a l so an integer. Moreover, the specification of r&, rj n and the function 
7„ : X n x y n — > (0, +00) depends on L(z), and will be discussed in the next section. 

As in [|TJ J2j Q, given any (x n ,y n ) G X n x y n , the performance of l n is measured by the number of 
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Algorithm 1 SA-LDPC-IED scheme X n 



1: Based on V1P2 • • -Vn and s 11 = H nxn x n , the encoder generates accumulated syndromes s\S2 ■ ■ ■ s n 

and augmenting syndromes a±a2 ■ ■ ■ a n . 
2: Based on V1P2 • • - V n and H„ xn , the decoder calculates matrices ^Axn^2Axn • • • H^ n . 
3: b <- 0. 

4: while The encoder does not receive bit 1 from the decoder do 

5: 6 •<— 6 + 1. 

6: if b < £ then 

7: The encoder sends augmenting syndromes a^-i)A+i " " " a &A to the decoder by A bits. 
8: else 

9: The encoder sends syndromes s' Vnn = H' Vnnxn x n to the decoder by rj n n bits. 
10: end if 

11: Upon receiving syndromes sent from the encoder, the decoder calculates x n by solving the 
optimization problem 

arg min zn:H (^ n= - 6A ln {z n , y n ) if 6 < £ 

argmin z n.H nxn2 n =s n iH ^ nnxn2 n =sl ,_ 7«(^ n ,y n ) otherwise 



f n = 



12: if 7„(x n |y n ) < T 6 or 6 > f then 

13: The decoder sends bit 1 to the encoder. 

14: else 

15: The decoder sends bit to the encoder. 

16: end if 

17: end while 

18: Upon receiving bit 1 from the decoder, the encoder sends s'^ n ^ +A = H" nH( -^ +A ^ xn x n to the decoder. 

19: Upon receiving s'^ K ^ +A , the decoder calculates the set 

D = {z n : n>[ nH{e)+A)xn z n = s'^ {e)+A ,wt{z n -x n )<e or wt(z n - x») > 1 - e} . 

If D contains a unique element x n , the decoder outputs x n as the estimate of x n . Otherwise, decoding 
failure is declared. 
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bits per symbol from the encoder to the decoder rf(x n ,y n \I n ), the number of bits per symbol from the 
decoder to the encoder ri ! (x n , y n \I n ), and the conditional error probability P(I n \x n ,y n ) of X n given x n 
and y n . Let j{x n ,y n ) be the number of interactions at the time the decoder sends bit 1 to the encoder. 
It follows from the description of Algorithm [T] that 

, j(x "f )A + H(e) + £ if j{x n ,y n ) < A/n 
r f (x n ,y n \l n ) = { n W " ■> y ) — (31) 

1 + T}n + H(e) + A otherwise 

r 6 (x",^|X ra ) = J(x "^ n) . (3.2) 
n 



and 



Moreover, let (X, 1") = {(Xi,Yi)} c ^ 1 be a stationary source pair. We further define 

r f (In) =E[r f (X n ,Y n \l n )} 
r b (l n ) ^E[r b (X n ,Y n \l n )] 

and 

P e (l n ) ^Pr{X n ^X n } 

IV. Performance of SA-LDPC-IED: General Case 

This section is devoted to the theoretical performance analysis of our proposed SA-LDPC-IED scheme 
T n for both individual sequences x n and y n and stationary, ergodic sources. Throughout this section, we 
assume that A ~ ^/n. 

A. Specification of j n (-, ■), r/ n , and {Tb}, and Probability Bounds 

In order for our proposed SA-LDPC-IED scheme T n to be truly universal, i.e., to achieve good 
performance for each and every individual source and side information pair (x n , y n ), we associate j n (-, •) 
with a classical universal lossless code C n (with block length n and the side information available to 
both the encoder and decoder), where C n is a mapping from X n x y n to {0, 1}* satisfying that for any 
y n G y n , the set {C n (x n ,y n ) : x n G X n } is a prefix set. Specifically, we define 

ln (x n ,y n ) = h n (x n \y n ) 

where nh n {x n \y n ) is the number of bits resulting from applying C n to encode x n from X given the side 
information sequence y n from y available to both the encoder and decoder. 

Following the approach adopted in j2j Q, it is essential to calculate the following probabilities 

Pr {H' Vnnxn x n = 0"""}, Pr {H' ( ; H(e)+A)xn x" = 0" H W+ A } and Pr {h^z" = bA } for 1 < b < 
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given x n / O n . In addition, in our case, the specification of r] n , and {1^} is also related to the 
probability Pr |H^ n x n = 6A j. Since H^ nxn and H" nH ^ +A ^ xn are obtained from Gallager parity 
check ensemble, it can be easily shown that 

Pr{H^ BXB x» = 0^} = 2"*» 

Pr{H^H (e)+ A)xn^ = 0" H(e)+A } = 2- H W- A 

for any x n / n . However, calculating Pi ^H.f A ^ n x n = bA j is much harder. 
It can be seen that 

Pr{H^ = 0^} 

depends on the support set of x n , i.e., the positions of non-zero elements in x n . Let x(x n ) represent the 
support set of x n , and we write x(x n ) simply as k whenever x n is generic or can be determined from 
context. Let LP^^ be the matrix consisting of those columns of H raxri with indices in x. The degree 
polynomial of x, denoted by L"(z), is defined by 

i 

where Lfn is the number of columns with degree U within , ,. And define 

L 
i=l 

Now let 

42 = max {^2^ ^l" 1 - (bA - 2^ ^l" 1 ) , o} , 
il* = max 

42 = min{2^ bA l -6A,i? 2 2^ feA l- 1 }. 

To understand the meaning of {4a.}. > ^ et us f° cus on ^bA = {A&A,i}i=r By the binary tree repre- 
sentation in the previous section, 

= # of A bAjl s.t. A bA:l C {1 • • ■ i?m} and 4 (AbA t) = 2^ 6A 1 

4 2 2 = # of A bAjl s.t. A 6Ajl C {1 • • ■ R in } and d, (AbA t) = 2^ ^l-i 

42 = #of A 6A ,iS.t. A bA , l <Z{R 1 n + l---n} andd v{AbAi) = 2^ bA ^ 

42 = #of A 6Ali s.t. A 6Ail C{ J R 1 n + l---n}and4 (AbAi) = 2r io & 6A l- 1 
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Since the block length n is assumed to be a power of 2, it follows that 



^ = min (^-2N^l,i? 1 2N^U 
n y n J 

n [ \ n 

*f = ma X |i? 2 2N^l-2(2N^l-^, } 



( 4 ) 

n [ n 

and hence i = 1, 2, 3,4, all depend only on bA/n. 

We have the following result, which is proved in Appendix |A| 

Lemma 1. Let L(z) be a normalized variable node degree distribution from a node perspective with 
minimum degree l\ > 2. Let c&a = 2 _ l" log2 vl and g(r,k) =(l+r) k + (l — r) k for any t and k. Suppose 
H nX n is uniformly picked from ensemble 'H n u z y Then for any x n ^ with its support set x, 



p r JH (6A) r r ' 



bA } < expjnP (^,M~) + ^ln(nr) + ilnnP<(l - £) + 0(1)} 



where 



r =max{ -,min{r,/-r} 
n 



and for any — , Z anc? £ £ (0, Z], P (— , Z, £) « defined as 



n 



± -m e (e/z") -einr 



n 

t (2) 
+ ^ A -ln 
n 

# (3) 

+ ^ A -ln 

n 

t (4) 
n 





nc b A) 




2 




2ric feA ) 




2 








2 




2r 2 c b A) 
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in which r is the solution to 



nc bA 



4a 9{T,ncbA - 1) 



n g(r, ric ftA ) 



-2ric b A 



-J*2Q, A 



(2) 

*bA 9{r, 2ric bA ~ 1) 
n g(r, 2nc 6A ) 

4a 9(T,r 2 c b A ~ 1) 
n g(r, r 2 c feA ) 

,(4) 



-2r 2 c bA 



*bA g( r ' 2r 2Q,A - 1) 

n ff(r, 2ric 6A ) 



(4.2) 



forte 



0, Z - %f vr (c 6A n) - %^tt (c 6A r 2 ) 



bA 



n 



oo 



(4.3) 



for £ € Z 



,00 

n 



7T (c 6A n) 



^^(c feA r 2 ),/ 

.(1) .(3) 



with the convention that e 



0. 



Remark 1. When £ = I — ^ir (q,a?"i) — ^r^r (c^a^), the solution r fo (4.21 is r = +oo. this case, 



the expression in t \4. i| ) should be understood as its limit as r — >■ +oo, i.e., 
'6A 



P 



-ZH e (£/Z~) + lim 



+oo 



*lnr | ^ ln g( r ' r i c ^) , 4a ln gMriC6A) 
n 2 n 2 



4a ln g(r, r 2 c bA ) 4a i gOj 2r 2CfeA) 
n n 2 n n 2 



,(1) t (3) 

-ZH e (£/Z) + — vr(c 6A n) ln[c 6A ri] + -^7r(c 6A r 2 ) ln[c bA r 2 ] 



when £ = 2 







TTlCfeAn, 
6A 



*(3) 
_ 



n 



71" (CftA^j. 



Remark 2. Replace ^ by any real number R E (0, 1] in i = 1,2, 3, a«<i 4, q,a» am^Z P U £,)■ 
is «of Ziara? fo verify that i = 1,2,3, and 4, q,a> P (R>li£) as a respective function of 
R G (0, 1] are aZZ weZZ defined. One can further verify that as a function of R G (0, 1], f/ie following 
identities hold: 

£ ^ = a (4.5) 



(4.4) 



and 



i=l 



,(2) 



n 



,(3) 



(i) 



HCfeA— + 2r lCfeA — + r 2 c 6A — + 2r 2 Q, A — 
n n n n 



(4.6) 
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As illustrated in Figure |3j the function P (P, Z,£) has several interesting properties including 
PR1 given (P, I), P (P, Z, £) is a strictly decreasing function of £ over £ G (0, Z/2]; 
PR2 given < £ < 1/2, P (-R, Z, £) as a function of P is continuous and strictly decreasing over 
R G (0, 1], and furthermore 

p (0, £) = lim P (P, Z", £)=0 

PR3 and P (P, I, £) is close to —Pin 2 when £ < Z/2 is not too far away from 1/2. 
These and other properties of P (P, I, £) are needed in the performance analysis of our proposed SA- 
LDPC-IED Scheme X n . Their exact statements and respective proofs will be relegated to Appendix IB] 




Fig. 3 . Graphical Illustration of P ( ^ , I , £) 



Based on the function P (^f ,Z,£), we are now ready to specify r\ n and {T^} for any 1 < b < ^ in 
our proposed SA-LDPC-IED Scheme Z n , which are defined respectively as 



T] n = 1 + 



In 2 



z — \ 3{l] , nl 1 reZ 



+ 



and 



In 2 



Lhe 



n 



311] nl 1 nl 

— — In In — 

A 2 2n 4 



A 

n 

A 

n 



where e > is the same as in the description of the SA-LDPC-IED Scheme Z n . 
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B. Performance for Individual Sequences 

We now analyze the performance of the SA-LDPC-IED scheme X n in terms of the performance of 
the classical universal code C n for any individual sequences x n and y n . We have the following theorem, 
which is proved in Appendix [C] 

Theorem 1. Let L(z) represent a normalized variable node degree distribution from a node perspective 
with minimum degree l\ > 2. Then for any (x n ,y n ) G X n x y n , 



r f (x n , y n \l n ) < R[f (e, h n (x n \y n )) + H(e) + 



:(*) 



r b {x n ,y n \Z n ) = 



2A 



n 



and 



P e (l n \x n ,y n ) < 2 - A + 1 °s 2 (i+ 1 )+°( 1 ) 



(4.7) 
(4.8) 

(4.9) 



where P e (I n \x n , y n ) denotes the conditional error probability ofX n given x n and y n , and ^l(z) ( e ' h n {x n \y n )) 
is the positive solution R to 

-P(R,l,he) 

ifh n {x n \y n ) < r», and 



K{x n \y n ) 



A 



n 



3R1 nl 1 nl 

ln2 + -V-ln 1 In — 

A 2 2n 4 



(4.10) 



R ( L %(e,h n ( X "\y")) = 2 + ± 



„ r , \ 3 TZl , nl 1 ni 



(4.11) 



otherwise. 



In order to analyze the asymptotical performance of the SA-LDPC-IED scheme X„ first as n — > oo 
and then as the average degree I of L(z) goes to oo, we define for any h E [0, 1] 

(A) 



and 



r L( z ) (e, h) =R L{z) (e, ^t) + H(e) - h. 



Clearly, ru z \ (e,h) represents the redundancy of Z n , i.e., the gap between the asymptotical total rate of 
X n and the desired rate h. We have the following two results, which will be proved in Appendix [Pj 

Proposition 1. Let L(z) be a normalized degree distribution with l\ > 2 and e be a real number where 
j-ljv < e < 0.5. Then for any h > 0, 



r L{z) (e,h) <H(e)+(l + l(/ i ln2 > -P(l,l,he))) I ^ exp 



In 2 



2Zie 



(W-i) 



1 / 2Zie.- 

+ hT2 exp 
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where I(-) is the indicator function such that 

\ 1 ifh\n2> -P(l.Uie) 
I (ft In 2 > -P(l,/,/ ie )) = I ~ V ; 

i otherwise 

Proposition 2. Let L{z) be a normalized degree distribution with l\ > 2. Then 

2_ 

for any k > e'i a«<i h > 0. 

C. Performance for Stationary, Ergodic Sources 

In this subsection, we analyze the performance of the SA-LDPC-IED scheme Z n for any stationary, 
ergodic source-side information pair (X, Y) = {(Xi, Yi)}^l 1 with alphabet X x y. To this end, we select 
{C n }^ c L 1 to be a sequence of universal (classical) prefix codes with side information available to both 
the encoder and decoder such that 

lim h n {X n \Y n ) = H(X|Y) with probability one (4.12) 

for any stationary, ergodic source-side information pair (X, Y). (Note that from the literature of classical 
universal lossless source coding (see, for example, (3j, |l4j, p3| , p6| , 1 17 1, and the references therein), 



such a sequence exists.) To bring out the dependence of X n on L{z) and e, we shall write Z n as I n (L(z),e). 
Then we have the following result, which is proved in Appendix [D] 

Theorem 2. Let L(z) be a normalized variable node degree distribution. Then for any stationary, ergodic 
source side information pair (X, Y), 

lim lim Tt ( X n ,Y n l n ( L(z k ), ) ) = K(X\Y) with probability one (4.13) 

x -(^))-°(^) 



n [ X n ,Y r 

and 



P e (l n (L(z k ), < 2 -A+iog 2 (t+i) + 0(i) (4 15) 



whenever k > 9. 
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V. Performance of SA-LDPC-IED: Binary Case and Bit Error Probability 

Theorems [T] and [2] show the performance of our proposed SA-LDPC-IED scheme T n in terms of 
the forward and backward rates versus the word error probability for both individual sequences x n and 
y n and stationary, ergodic sources. In this section, we consider instead the forward and backward rates 
versus the bit error probability by focusing on independent and identically distributed (i.i.d) source-side 
information pairs (X, Y) = {(Xj, Y^)}?^, where the source X and side-information Y are correlated 
through a binary symmetric channel with a cross-over probability po G (0, 0.5), which is unknown to 
both the decoder and the encoder. Limiting ourselves to this smaller class of source-side information 
pairs allows us to illustrate the SA-LDPC-IED scheme X n by using a specific and simple function •), 
which in turn leads to further simplification of the SA-LDPC-IED scheme T n itself and paves the way 
for the belief propagation (BP) decoding to be used as a decoding method in IED in the next section. 

Note that in this binary case 

R(X\Y) = H(po) . 

Define H _1 (-) : [0,1] -»• [0,0.5] as the inverse function of H(-) such that x = H x (/i) if and only if 
h = H(x) for x G [0,0.5] and h G [0, 1]. Now specify 7(-, •) as 

l -^±±+R(±wt(x n -y n )) if ±wt(x n - y n ) < 0.5 



i{x n ,y r ' 



(5.1) 

i + 1 otherwise. 



it 



It is easy to see that j(x n ,y n ) is actually the normalized code length function of the classical prefix code 
C n with side information available to both the encoder and decoder as described in Algorithm [2] With 
the assumption on the correlation between the source X and side information Y and with this specific 
function 7(-, •), we can further get rid of the last round of transmission from the encoder to the decoder 
in T n , yielding a simplified version X n as described in Algorithm [3] 

Now let us analyze the performance of the SA-LDPC-IED scheme T n in terms of the forward and 
backward rates versus the bit error probability Pb, where 



n 



wt(X n - X r 

Then we have the following theorem, which is proved in Appendix |E] 



Theorem 3. Let L(z) be a normalized variable node degree distribution from a node perspective with 
minimum degree l\ > 2 and average degree I being an odd integer. Select e > such that e < 0.5 — 
H _1 (0.75). Then for any i.i.d source-side information pair (X, Y) correlated through a binary symmetric 
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Algorithm 2 A classical prefix code C n with side information available to both the encoder and decoder 
1: The encoder calculates w = wt(x n — y n ). 

2: if w < 0.5n then 

3: The encoder sends bit followed by a codeword of fixed-length In n specifying w and then by a 
codeword of length nH f^J specifying the index of x n — y n in the set {z 11 : wt(z n ) = w} sorted 
by the lexicographical order. 

4: else 

5: The encoder sends bit 1 followed by x n itself. 
6: end if 



channel with cross-over probability po £ (0, 0.5) and for sufficiently large n, 



(5.2) 

r b (X n ) = o I 4= J (5-3) 



and 

Pb{in) < € + e - 2 "(°- 5 -P°) 2 + 2 -A+log 2 ( i+ l)+0(l) _ (5 4) 



By defining 



r L (z)(e>Po) =Rl{z) (e,H(p )) - H(po) 



we have the following proposition, the proof of which is omitted due to its similarity to that of Proposition 

m 

Proposition 3. Let L(z) be a normalized degree distribution with l\ > 2 and k > 2. For po G (0, 0.5), 

r^,(^.».)= (.-^i")- 
We conclude this section by providing the following theorem (proved in Appendix [F]), which analyzes 
the performance of the modified SA-LDPC-IED scheme X n when L(z k ) is used. Once again, to bring 
out the dependence of X n on (L(z), e), we write l n as l n (L(z),e). 

Theorem 4. Let L{z) be a normalized variable node degree distribution with minimum degree l\ > 2. 
For any i.i.d source-side information pair (X,Y) correlated through a binary symmetric channel with 
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Algorithm 3 SA-LDPC-IED scheme X n for i.i.d source-side information pairs 
1: Based on V1P2 • • - P n ar >d s n = H nxn x n , the encoder generates accumulated syndromes s\S2 ■ ■ ■ s n 

and augmenting syndromes a\a,2 ■ ■ ■ a n . 

2: Based on V1P2 • • • V n and H nxn , the decoder calculates matrices H^^H^^ • • • H^ n . 

3: b^O. 

4: while The encoder does not receive bit 1 from the decoder do 

5: 

6: if b < £ then 

7: The encoder sends augmenting syndromes a^-i)A+i " " " a feA to the decoder by A bits. 
8: else 

9: The encoder sends syndromes s' VnJl = H.' Vnnxn x n to the decoder by rj n n bits. 
10: end if 

11: Upon receiving syndromes sent from the encoder, the decoder calculates x n by solving the 
optimization problem 



x n = < 



arg min z „ :H (^„ =SbA 7n (z n , y n ) if b < I 

arg min J? „.H nxn a»= a » > H'„ BnXB z»=^ Bn Jn(z n , y n ) otherwise. 



12: if ~f n (x n \y n ) < T b or b > f then 

13: The decoder sends bit 1 to the encoder, and outputs x n as the estimate of x n . 
14: else 

15: The decoder sends bit to the encoder and leaves the estimate of x n undecided. 
16: end if 
17: end while 



cross-over probability po G (0, 0.5), 



and 



&Jt r '( t ( I<,i) '55j)) -!W <5 ' 5) 

^'(^•^H^) <5 - 6) 
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whenever k > 



2(l-2p ) 



VI. Implementation and Simulation Results 

To verify our theoretical analysis in the last two sections, we have implemented our proposed SA- 
LDPC-IED schemes with some modification, namely by adopting the BP decoding in the place of the 
minimum coding length or Hamming distance decoding. In this section, we report their performance for 
binary source-side information pairs (X,Y), where X and Y are correlated through a binary channel 
with probability transition matrix (from Y to X) given by 

1 - Pi P2 
Pi 1-P2 

and where pi,p2 £ (0,0.5] are assumed unknown to both the encoder and decoder. Since the standard 
BP decoding algorithm applies only to fix-rate LDPC codes with known statistics of source and side 
information pairs, we first have to modify the BP decoding algorithm so that it fits into our variable-rate 
and unknown statistics situation as well while maintaining its low complexity. 

A. Modified BP Decoding Algorithm 

The BP decoding algorithm can be considered as a sum-product algorithm |T8j on a Tanner graph, 

which represents the parity check matrix of the LDPC code, with variable nodes corresponding to bits 

of the source, and check nodes corresponding to syndromes. Generally speaking, it tries to marginalize 

the distribution of each bit of the source based on local calculations. Specifically, it iteratively calculates 

messages from variable nodes to their connected check nodes, and vice versa, i.e. 

i Pr{X t = 0\Yj} t ^ 
m v ^ Cj = ^ Fv{Xi = m} + L rn Ck ^ Vi (6.1) 

c k ^cy.c k is connected to «, 



m Cj -> Vi = 2tanh~ 1 (l - 2s j) J] tanh ^"p 3 ) (6-2) 

v k ^Vi-.v k is connected to 

where m Vi -+ c . and m c ._» Vj are messages passed from the variable node Vi to the check node Cj and vice 
versa, respectively, and Sj is the syndrome corresponding to Cj. After certain iterations, assuming the 
calculation converges to a stationary point, the marginal distribution of each variable node is calculated 
based on the messages sent from its connected check nodes, and the decision on each bit is made 
according to the distribution in the following way 

U 11 Pr{Xi = l\Y t } + 2^ "T-Cfc^t;, di U 

c k -.c k is connected to u, (6.3) 

1 otherwise. 
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To initialize the iterative procedure, for each variable node Xi, the marginal distribution is assumed to 
be (Pr{Xj = 0|yj},Pr{JQ = 1|^}). Therefore, the standard BP decoding algorithm needs the statistics 
of source and side information as inputs. 

However, in our case, the statistics of source-side information are unavailable, i.e., p\ and p2 are 
unknown. To deal with this problem, let us first consider the case pi = P2 = Po, i.e. X and Y are 
correlated through a binary symmetrical channel. Now let 

Inn + 1 



Pb = H 1 ^max |o, Tj, 



n 

where p b can be interpreted as the maximum cross-over probability of the binary symmetrical channel 
correlating X and Y, such that the error probability of the SA-LDPC-IED scheme l n can be maintained 
asymptotically zero at the 6-th interaction. Therefore, we will use pb as the input to the BP decoding at 
the 6-th interaction. Moreover, at each interaction, decoding failure is detected and the decoder will send 
bit to the encoder for more syndromes if one of the following two situations occurs: 

• the number of bits with significant log-likelihood (larger than certain value) is less than a threshold 
within first several iterations of BP decoding; 



• or the number of syndrome constaints satisfied by the codeword calculated using ( |6.3[ ) at the end of 
each iteration does not increase for several iterations. 
On the other hand, successful decoding is identified when the modified BP decoding algorithm converges 
to a codeword satisfying all syndrome constraints without encountering those two situations listed above. 
Simulation shows that under this decoding rule, the bit error probability is still very small. Moreover, 
since this decoding rule is more aggressive than threshold decoding used in section |vj for some (X, Y) 
the rate achieved by the SA-LDPC-IED scheme implemented in this way can be smaller than that given 
in Theorem [3] 

To further consider a general memoryless source-side information pair, i.e. p\ ^ P2, at the 6-th 
interaction, we can quantize p\ into a quantized value, say q\, then calculate the quantized value q2 
of p2 according to 

Pr{y = 0}H(gi) + Pr{y = l}H(g 2 ) = H (p b ) 

and finally apply the modified BP decoding algorithm for each such quantized pair (qt,^)- Successful 
decoding is claimed whenever there is one such quantized (gi, 92) that makes the BP decoding algorithm 
converge to a source sequence satisfying syndrome constraints. When there is a tie, i.e. more than one 
pair (q\, 52) that make the BP decoding algorithm succeed with different outputs, we will choose the one 
with the smaller value of q\. Here we assume that the distribution of side information Y is known to the 
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decoder. Otherwise, the empirical distribution can be calculated, since the decoder has the full access to 
side information. 

B. Simulation Results 

We first consider the case where the source and side information are correlated through a binary sym- 
metrical channel with unknown cross-over probability, and the side information is uniformly distributed. 
Figure [4] shows the performance of our implemented scheme (referred to as the simulation rate) along 




Fig. 4. Performance of SA-LDPC-IED: Symmetrical Channel 

with the conditional entropy rate and the performance upper bound established in Theorem [3] where the 
blue solid line represents the simulation rate with bit error probabilities below or around 2 x 10~ 5 , and 
the green dashed line represents the upper bound established in Theorem [3] with e = 0.1. The block 
length is 8000, and the variable degree distribution (from an edge prospective) used is shown below: 

X(x) = 0.1787042 + 0.176202x 2 + 0.102845x 5 

+ 0.114789x 6 + 0.0122023x 12 + 0.0479225a; 13 
+ 0.115911x 14 + 0.251424x 39 
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which is designed for rate 0.5, and obtained from |19j . It can be seen that our implemented SA-LDPC- 
IED scheme can indeed adapt to the entropy rate H(X|y) well in a large rate region. To interpret the 
upper bound R L ^(e, H(po)) also shown in Figure |4] better, an explanation on e is needed here. The 
reason that e >> bit error probability in the simulation is due to the minimum hamming distance d^ in 
of the code generated by H^^ n . From the proof of Theorem [3J it follows that with high probability, 
±wt{X n -X n ) < e. On the other hand, ±wt(X n -X n ) < e implies that X n = X n if d®^ > en when the 
coding procedure terminates at the 6-th interaction. Moreover, since the implemented decoding algorithm 
only checks syndrome constraints to determine the decoding success, instead of using thresholds given 
in Theorem [3] the bound on rate can be improved if the choice of e for the 6-th interaction depends on 
^mL> especially for the high rate case as d^ in increases with 6. However, since d^ in can not be expressed 
in a neat way and does not affect redundancy with respect to k when L(z k ) is used, we do not include 
the corresponding result in this paper. In the meantime, by using the same degree distribution L(z) in 
Figure Q Figure jsj shows how fast RL( z k ) (^Tp^Pci)) conver § es to H(j>o)> where the gap is always 
less than 0.02 when k = 5. 



0.05 




0.4 0.6 
Entropy Rate H(p ) 



1.0 



Fig. 5. Redundancy bound with different k 



We next consider source and side-information pairs correlated through binary asymmetrical channels. 
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Table [T] lists our simulation results, where the side information Y is still assumed to be uniformly 



Pr{X = 1\Y = 0} 


Pr{X = 0\Y = 1} 


Rate 


0.05 


0.1959 


0.541 


0.1 


0.1206 


0.544 


0.15 


0.0766 


0.543 


0.2 


0.0481 


0.540 



TABLE I 

Performance of SA-LDPC-IED: Asymmetrical Channel 

distributed, and the transition probabilities are selected such that H(X|Y) = 0.5 for all cases. In our 
simulation, we did not see any error in 1000 blocks, each block being 8000 bits. As can be seen, our 
implemented SA-LDPC-IED scheme also works very well in this situation too. 

To make a comparison with SWC, a SWC scheme using the same LDPC code (LDPC-SWC) was 
also implemented for the source and side information correlated through a binary symmetrical channel. 
The respective results are shown in Table InJ where bit error probabilities are maintained below 10 -5 for 



H(X\Y) 


^SA-IED 




0.426 


0.473 


0.5 



TABLE II 
SA-LDPC-IED VS. LDPC-SWC 



both SA-LDPC-IED and LDPC-SWC schemes. Note that i? sw is deliberately chosen to be 0.5, since 
the degree distribution of the LDPC code used here is designed for rate 0.5. Moreover, in the simulation 
of the LDPC-SWC scheme, we assumed that the cross-over probability po is known to the decoder, 
while in our implemented SA-LDPC-IED scheme, po is unknown. Clearly, simulation results show that 
SA-LDPC-IED outperforms LDPC-SWC. 

VII. Conclusion 

In this paper, interactive encoding and decoding based on binary low-density parity-check codes 
with syndrome accumulation (SA-LDPC-IED) has been proposed and investigated. Given any classical 
universal lossless code C n (with block length n and side information available to both the encoder and 
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decoder) and an LDPC code, we have demonstrated, with the help of syndrome accumulation, how to 
convert C n into a universal SA-LDPC-IED scheme. With its word error probability approaching sub- 
exponentially with n, the resulting SA-LDPC-IED scheme has been shown to achieve roughly the same 
rate performance as does C n for each and every individual sequence pair (x n ,y n ) and the conditional 
entropy rate H(X|y) for any stationary, ergodic source and side information (X,Y) as the average 
variable node degree I of the underlying LDPC code increases without bound. When applied to the 
class of binary source and side information (X, Y) correlated through a binary symmetrical channel with 
cross-over probability unknown to both the encoder and decoder, the SA-LDPC-IED scheme has been 
further simplified, resulting in even improved rate performance versus the bit error probability when T 
is not large. Coupled with linear time belief propagation decoding, the SA-LDPC-IED scheme has been 
implemented for binary source-side information pairs, which confirms the theoretic analysis, and further 
shows that the SA-LDPC-IED scheme consistently outperforms the Slepian-Wolf coding scheme based 
on the same underlying LDPC code. In the course of analyzing the performance of the SA-LDPC-IED 
scheme, probability bounds involving LDPC have been established, and it has been shown that their 
exponent as a function of the SWC coding rate, the average node degree I, and a weighted Hamming 
weight of a codeword has several interesting properties. It is believed that these properties can be applied 
to analyze the capacity-achieving performance of LDPC for channel coding as well, which will be 
investigated in the future. 
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Appendix A 
Proof of LemmaQ] 

We consider only the case in which I is not an integer. The case where / is an integer is a bit easier 
and can be dealt with in a similar manner. 

Although there is thorough analysis of the probability Pr {H mxn x n = m } for H mxn from T-L m ,n,L{z),R{z) 
in 1 20 1, 1 21 1, |22|, and |23| , the result therein in general is not applicable to H^^ n , the matrix 
obtained from syndrome accumulation on H nxn . Towards analyzing Pr {h^^x™ = bA |, we focus on 



{P&Aj&Li defined in section III-A Given VbA = {A&A,i}i=i» one can classify AbA,i into three categories: 

• A 6 A,i Q {1,2, . . . ,Rin}, 

• A-bA,i Q {R\n + 1, R\n + 2, . . . , n}, or 
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• A 6A ,i £ {1, 2, . . . , i?in}, and A feA ,i + 1, Rin + 2, . . . , n}. 

To avoid complicating the analysis unnecessarily, we assume that there does not exist AtA,i falling into 
the third category. Further effort reveals that this assumption holds if and only if 2 T_ L lo g2 A J \R 1 n, or in 
other words, 

JD _ C 

1 2L lo g 2 Aj 

for some positive integer C, where the parameter A is a function of block length n. In fact, in this paper 
we only consider the case where A ~ y/n, which implies 2 L lo S2 A J and therefore the assumption 

above always holds for sufficiently large n if I is a fractional number with a power of 2 as its denominator. 
Consequently, each AtA,i can be further categorized into one of four cases: 

• A 6 A,i C {1,2,..., Rw}, and |A 6A;i | = 2 T ~^ W ; 

• A 6Ail C {1, 2, . . . , Rm}, and |A 6A;i | = 2 T -^ bA ^+ 1 ; 

• AbA,* C {R in + l,Rm + 2,..., n}, and |A 6A)i | = 2 T -^ feA l ; or 

• A 6AjJ C {R in + l,R in + 2,..., n}, and |A 6A ,i| = 2 r "f lo & bA l+ 1 . 

Now we use {4a} to re P resent the number of A& A /s falling into each category, which are given by 
the following formulas: 

tlJ = min 

tfl = max {R 1 2^ bA ^ - (bA - 2^ bA ^ , } , 

42 = max{^ 2 2^ bA l -2(2^ 6A l -6A) ,0}, 

42 = mm{2^ bA ^ -bA,R 2 2^ bA ^ 1 \. 

Note that we assume that block length n = 2 T for some integer T. It then follows that 



,(2) 

n 

f (3) 



Recall that 



^ = min(^-2N^l^ 1 2^^U 
n \ n J 

^ = max |i? 2 2N^l -2(2N^l-^),o} 

^ = min( 2 N^l-^,i? 2 2N^l-i 
n [ n 

CbA = 2 T -^ bA ^ = 2-r iog ^i . 
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Therefore CbA also depends only on 

Now define % n ,L(z),*r,6A as a subset of H n u z ^ such that 

Hnxn ~ T~^n,L(z),x,bA 

if and only if 

H nxn E H n , L(z) and H^x" = bA 

where x is the support set of x n . It is easy to see that given x 11 (and therefore x), these subsets 
T~Ln,L(z),x,bA are nested with each other 

,x.bA 

if a > 6. Furthermore, let A„ >>CifeA = |^ n ,i(^),^,6Al- Tnen we have 

(h£3„s» = 6A ) = (A.1) 

where H nxn is uniformly picked from r rL n u z )- Therefore the main issue is to derive asymptotic formulas 
for \H n Uz)\ an( l h-n,L{z).x.bA- At this point, we invoke the following result from Mineev and Pavlov 



24} (see also [25] for a stronger version). 



Theorem 5 (Mineev-Pavlov). Suppose Tigris the ensemble of m x n 0-1 matrices with i-th row sum r% 
and j-th column sum lj satisfying maxjrj, lj : 1 < i < m and 1 < j < n} < log 1 / 4 ™ 6 m, where e is an 
arbitrarily small positive constant. Then 

(E™x~i)! 



(A.2) 



where < 5 < 0.5 is an arbitrarily small constant. 

First of all, applying Theorem [5] to I'H^l^)!, we have 

I^WI = < , 77^7 (CL(z)+o(n-°- 5+s )) 

(ri!)^-(r 2 )^-nf=iOiO-* n 

where 

= exp 



(/2in(n - 1) + i? 2 r 2 (r 2 - 1)) Ef=i " 1 



2P J 
Towards calculating A^x^^^A- note tnat ea ~h H nxn consists of two sub-matrices H* x . . and H^-i 
where x c is the complement of x. Suppose {rf r }™ =1 is the row-sum profile of H*. ,. Then the row-sum 



January 26, 2012 



DRAFT 



31 



profile {rf}f =1 of H£, , |} is given by 



(n-M) 

rf ° = ri — r% for 1 < i < R\n 
r*° = r 2 — rf for i?in + 1 < i < n 

For each H nxn G Un,L(z)> its ^nx\n\ anc * ^nx^-l^l) snou ld nave an d ^"(z) as tne i r column- 

sum profiles. Therefore 



< rf < n for 1 < i < i?m 
< rf < r 2 for i?m + 1 < i < n 

n 



n 



i=i 



Note that 



n 6Axn X 



\ zUeA bA SieA bA ^»,2 



\ SieA bA T^j&x hij / 



SieAi 



\ 



/ 1 \ 



\ SieA bA SieA bA • • • SieA bA ^Jm j 



V 1 / 



(A.3) 
(A.4) 
(A.5) 



\ SieA bA r i j 
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Then H nxri G H n ,L{z),^,bA if an d only if 

2 



< iAj+ u ^ < j < 42 - 1 

U=l 
2c bA 

E< 

C(,A 

El 



4A C bA+2Ci, A j+'ti 



for < j < tfl - 1 



u=l 
2c bA 

E r ,' 



4A C l>A+24ACbA+CbAi + M 



for < j < - 1 



1 A c b A + 2^ a Cb A +4 A C b A + 2 Ct, A j +« 



for < j < 4a " 1 



(A.6) 
(A.7) 
(A.8) 
(A.9) 



Let 7£ bA ^ denote the set of all row-sum profiles {r|*}™ =1 which satisfy the constraints ( A.3 1 to ( A.9 1. 
Furthermore, let Af andAf,, C1 „ denote the number of H 5 * , ,'s and H^' , , h 's with the given 
row profile {r*} n and {r* c } n , respectively Then it is easy to see that 



An,L(2),^,6A — X] ^flLi^r },™ = i 

{n}r =1 6^6A,r^ 

Applying Theorem |5j to and A^^ , we have 



(A. 10) 



A 



(nr=i^onti(y)^ 



(C r « + (n- a5+(5 )) 



(I 



(nr=i-ronii(y) Lfn 



(A. 11) 



where 

Similarly, 

A " 



exp 



r 2 {l L ~ 1) 



{^r}r=i 



< exp 



{(i-n 



2/* 



< C r « < 1 



n 



-(C r ~c +o(n-°- 5+s )) (A. 12) 



nf=T(n - rr) n^n+i^ - rr) ) nti (w 



where 



exp 



r 2 (>L - 1) 



< exp 



r2ZLi(Li-L?Mk-l) 
2(1 - I") 



< CU= < 1 . 
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Combining (A.l I with (A.10) to (A.12) yields 



phh<£L*» = o*' 



A 



n,L(z),x,bA 
l^n,L(^)l 



Sir 



C r »C r>e c (nl")\(n(l-l"))\ 



< 



< 



2 M'T}i=ie^A,~ nf = i(^!) Ll " nf=T rfKn-rf)! Tl? =Rin+1 r?\(r 2 -r?) 



c 



L(z) 



nl 



C 



L(z) \ nV 



(nQl 

e n' 



n 



r-2 



(A. 13) 



{r? }"=ie7?. b A,« i=1 \ r i / i=Rin+l \ r i 

for sufficiently large n. To further evaluate Pr{H[^ n x n = bA }, we define the type (mW, m^ 2 \ , m/ 4 )) 
of {r?Yi=\ as follows: 



m 



(l) 



S 5 r ^Aj+« " s for < s < c 6A ri 

j=0 \m=1 / 



rn 



(2) 



■???■ 



(3) 



m 



(4) 



4a"1 /2c 6A 

E 5 E 

j=0 \u=l 

E ME 

j=0 \u=l 

^ 4 2"1 /2c 6A 

E { E 

j=0 \u=l 



(1) - s \ for < s < 2c b A^i 

CbA^A+2CbAj+« / 



Tad , „ + (2) , - s for < s < c&a?"2 

*6A+2*bAC 6 A+CbAj+« / 



rf(i) , _. ( a) , .(3) , „ - s for < s < 2cfeA^2 

t 6 AC 6 A+2t 6A Ci.A+t 6 AC 6 A+2c b Aj+« / 



where 



<*0«0 = < 



1 if x = 
otherwise. 

Now we can see that {r|*}™ =1 belongs to HbA,* if an d only if its type OmS 1 ', m( 2 \ m^) satisfies 



in 



(i) 

2j 



E 

3=0 

CbA^l 

j=0 

|^£6AI2 | 

E 

3=0 

E(4) 
m 



m 



(3) 
2j 



2.7 



£ {1) 



t {2) 



,(3) 



£ {4) 
l bA 



(A. 14) 
(A. 15) 

(A. 16) 
(A. 17) 



j=0 
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and 



I ^iATl I 

L 2 J <-bA ' ] 

E -i ■ m i ■ E 



: tA r 2 



' 1 L 2 J Cj, A r 2 

] 2j ■ mg } + ]T 2j • mg ) + E 2j ■ m$ = T~n 

=0 7=0 7=0 



(A. 18) 



Denote the set of types {m/ 1 ), rrS 2 \ rr?/ 4 )} satisfying the above constraints (A.14i to ( A.18 1 by 
MbA,x- If MbA,* 0, then the constraints ( |A.14[ ) to (A.18) implies 



< 

L 2 J c bA ri 

y~] {cbAn - vr(cfeA^i) - 2j) m 2j + E ( 2cfcAri ~ 2 i) m 2j 

j=0 j=Q 

I <=i,A r 2 I 

L 2 J c bA r 2 

+ E ( C ^ r2 ~ ^(cfcA^) - 2j)m^ + ^ (2c 6 Ar 2 - 2j')m^ 
j=o j=o 

= 42(^^1 - 7r(Q,An)) + 2t^C6A^i + 4a( c &A^2 - vr(c fe A^2)) + 2t^ c 6Ar2 - V*n 



= nl - 4A 7r ( c 6A^l) - 4a 7F ( c 6A^2) - 



77 



(A. 19) 



and therefore 



Jx < [_ J=W(<% A ri) - -^i(c 6A r2). 
n n 



On the other hand, Mt,e = implies Pr ^H-^Axn^ = feA | = 0, and hence the lemma is proved when 

,(1) + (3) 



Now suppose 



For convenience, define 



r > i- ^(0^) - ^{cbAT2). 

n n 

f W f (3) 

< I- ^{cbAn) - ^(c bA r2). 
n n 



fc( 3 ) 
k^ 



CbAn - ^(cbAri) 

2 

CbAn 

CfeA^2 - 7r(c fe A?-2) 
2 

CfcA^2 
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To proceed, we can group {r*Yi=\ w i tn tne same type together, and therefore have 

Rin 



e n 



ri 



n 



r-2 



{rf }" =1 e7e bA ,« *=1 \ 'i I i=Rin+l 



t 



(i) 
6A 



n i 

jm( 1 ),m( ! ),ra( ! ),m(*)}eM l , 4i «i=l \ m ' m 2 ' • • • > m 2fc« 



e n 



'"1 



2J fc< 2 > / 

n e n 



Q,A 



''2 



fc< 3 > / 

n e n 

j=0 \{r^}° bA 1 :X;?ii^=2j«=l \ r « 



2c, 



''2 



2J k<*> / 

n e n 

j=0 Vjr^} 2 ^:^^! r*=2j M=1 V T u 



Now define for any j > 



.(1) A 



C(,A 



e n 



(2) A 



2c bA 

e n 

| r «| 2c 6A .^ 2c i,A r >c—j U = l y 



(3) A 

j 



r-2 



e n 



(4) A 



Furthermore, we define 

M {m*}U =n 

Therefore 



''2 



2c bA 

e n 



1=1 



w 

6A 



(0 



fc(') (i) 

n 



Rin 



e n 



n 



r-2 



M {m^}U 

{m«}* =1 6M iA] x 



In view of (A.14i to (A. 18 1, we can get a trivial bound on |.M&a,k| as follows: 



\M 



bA, 



n/ 5 



+ 1 



fc< 2 > 



nV 



+ 1 



fc< 3 > 



n/ 5 



+ 1 



< 



< 



3[T| CbA 
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In a similar manner, in view of ( A.14 i to ( A.17 i and ( A.19[ we have 

\M bA ,„\<(n(l-n) mCb& 



Define 



Then we have 



r = max<j -,mm{r,l-r} } . 



E M ^nu 



'3n\T\ 



max Mr (j)1 4 



< exp 



6A 



ln(nP 



max M (m( ,)}4 



where the last inequality is due to the fact that q>a < This, coupled with (A. 13 1, implies 
Pr{H^ B *» = 0"} 



-i 



< exp 



3nff| 
6A 



ln(nr) + 0(1) 



max M fmW ,4 . (A.20) 



To continue, we now upper bound 



max Mr m(jl} 4 



under the conditions (A.14) to (A.18). By the type bound [26 Lemma 2.3], 



max In Mr ( 



{m«}* =1 



max 



n e 



1 \llj=0 m 2j • j=0 

4 4 feW 4 JfcW 



< max <; 53 *S * *S - E E ( ™ S- ln ) + E E ™£- ln $ 



i=l j=0 



i=l j=0 



< 



maxG ( |m^| 



(A.21) 



where 



4 fc< ; > 



G ( F° L ) = E 41 in 41 -EE K? ln 40 + E E ™g ln $ (A.22) 

V " - i=l i=l .j'=0 i=l j=0 



in which m 2 - can take any non-negative real number with constraints (|A.14|) to (|A. 18|). Since the function 



f(x) = —x ln x + cx 
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is concave in the region x > 0, it follows that G (^{ m ^}^ =1 ^j is a concave function, and hence the 
maximum can be calculated by using K.K.T condition, which is shown as follows. 
Define the function F ({m- ( ' i - ) }^ =1 , { a i}i=i>/^ as 

V ' V *~ ' i=l j=0 i=l i=0 

Now by taking the derivative of F (^{m^}^ =1 , {ai}f =1 , with respect to we have 

r\ T—i 

ln/z/V; 1 • ln^V • n; • 2.y 1. 



om 2 - 

According to K.K.T condition, let this derivative be zero, and we have 



Since 



it follows that 



For convenience, define 



Then 



which implies 



m 2j - e ?2j 



3=0 



j=0 



<7 W (r)=E^ 



gOi-l _ ''bA 



5 W( e /3) 

Now by taking into account the condition 

4 fc<*> 



5 «( e /3) 



EE 2 ^g = ^ 

i=l j=0 



we have 



It is easy to see that 



4 Ji) fc<» 
J"=0 

fcCO 



E^E 2 ^M- 



E2ir 2 ^«=r ff '«(r) (A.24) 

j=0 
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where 



Therefore is the solution to 



g' (l) (T } 



dr 



2 -/ 



(i) e K 5 



i=l 



l"n 



Putting (A.22) to (A.25) together yields 



max 



V 7 8=1 

4 

= £ 



fc (i) (i) 



1 



i=l 



Substituting e@ by r, we have 



max ( 



* i=l 



r) -Tnlnr 



where r is the solution to 



Notice that 



2 > bA 



W r 2 



/(0( T ) 



((1 + 



C6A / ri 

nil, - 

u=l \r*'=0 \ "u 



i=o 



Meanwhile 



Therefore 



(1-r) 



C(,A»"l 



<7 (1) (r 



j=0 

Ei 1} (-) 2J 

3=0 

(1 + r ) c " ari + (1 - r) CbAri 



5(r, c bA n) 
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where g(r, k) is defined in the lemma. Similarly, we can show that 



5 (3) (r) 



5 (4) (r) 



9(t, 


2c bA n) 




2 




C bA r 2 ) 




2 


g(r, 


2c bA r 2 ) 



It is not hard to verify that 



, (l) Tg'(T,c b Ari) (2) Tff'(r, 2cj, A n) (3) rg^r, c feA r 2 ) (4) Tg'(r, 2c bA r 2 ) 
bA g{r,c bA n) + bA fltr^QAn) + 6A g(r,c bA r 2 ) + bA 5(r,2c 6A r 2 ) 

= nl- t^n ^^-V - 2tg CbA n ff(r ' 2cbAn " 1} 



g(r, c 6A n) 

,(3) g{T,c bA r 2 - 1) (4) 

-*6 A c& A r 2 ? r 2tl^c bA r 2 

g{T,c bA r 2 ) 

which, together with ( A.26[ ) and ( A.27| ), implies 

max G |m (i) j 



g(r, 2c bA ri) 
g(r, 2cj, A r 2 - 1) 
#(r, 2c bA r 2 ) 



-nrinr 



+ ^ln 



#0, 


HCb A ) 




2 




2ric ftA ) 




2 




?~2Q>a) 




2 




2r 2 c bA ) 



(A.28) 



where r is the solution to 



nc bA 



4a g( T ,nc b A - 1) 



n g(r, ric 6A ) 



+2ric bA 



-^2Cf, A 



-2r 2 c bA 



(2) 

*bA ff( r ' 2ric bA - 1) 
n g(T,2nc bA ) 

4a ff( T > r 2Q>A - 1) 
re #(t, r 2 c feA ) 

4a s( r > 2r 2 c bA - 1) 



n g(r,2ric 6A ) 



(A.29) 
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Putting (|A.20[), (|A.21|), ([A.28]>, and (|A.29|) together, we then have 



PHHiS/ = o^' 



6A 



< exp < max G 




max Mr m( , n 4 



{» (i) L 



1 +^1 ,„(„,-) + 0(1) 




£ <<xp <j -///II e ^y^j + ^lnn/ 5 * ^1 - y^j +maxG 

rxi) ^ „ P ( ^, r, lA + ^1 ln(nF) + l - lnnf* (l - y ) + <>( ' > 



where the last inequality above is due to the fact that 

-l 

nl \ _ . ._, J 



in(nr) + o(i; 



In 



n/ 5 



< -nlH e (! x /l) + - lnnr ( 1 - y ) + 0(1) 



which can be derived from Sterling formula. This competes the proof of Lemma [j] when I* < I — 

t {1) , s t l3) , 

Finally, let us look at the case when V* = I — — 7t(c&a?"i) — —*n{cbAT2)- In this case, it follows from 



(A. 19) that M.tfi contains only one type, i.e., the type given by 



rn 



(0 



*0 



2fc« 



t w if n 
z bA 11 J 

otherwise 



(A. 30) 



for i = 1, 2, 3, and 4. Combining this with ( A.21 1, one can verify that in this case 

maxlnMr m(l) |4 =i = ij^^An) ln[o, A n] + 4 a ^(^A^) ln[o,A?"3] • (A.31) 
Plugging (A.31) into (A.20l then leads to the desired result. This competes the proof of Lemma [T] 



Appendix B 
Properties of P [R, I, £) 

This Appendix is devoted to several lemmas related to the function P (i?, /, £J , which are needed in 
our performance analysis. To keep our notation consistent as in Lemma [TJ only R = ^ appears explicitly 



in the statements of these lemmas. However, in view of Remark |2j (4.5), and (4.6 1, by replacing ^ by 
any real number R £ (0, 1], all lemmas in this appendix (Lemmas [2] to [6]) remain valid. Their respective 
proofs are the same whether or not R G (0, 1] is in the form of R 



bA 
n ' 
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In view of (|4.2|), we define 

A 



I 



bA 



n 



,l,T 



4a c 6A^i g(r, c bA ri - 1) 2t^CbAH #(t, 2c&An - 1) 



" g(T,2c b Ari) 
tflc bA r 2 g(r, c bA r 2 - 1) 2*^0^2 5(r, 2c feA ^2 - 1) 



g(r, 2c feA ^2) 



n g(r,c bA r 2 ) 
Lemma 2. Given ^ a«c? the following properties hold: 

PI As a function of r, r) strictly increasing over the interval [0, +oo). 

P2 For a/ry € [0, £ — -^f-vr (Q,A r i) — "TT 71 " ( c fcA r 2))> ffere ij a unique solution of t to f (— , t) = 

Proof of Lemma j^J In view of the definition of f(~,r, r), for Property PI, it is sufficient to prove 
that 9 ^fc^ as function of r is strictly decreasing over r G [0, oo) for any positive value k > 1 . To this 



end, take the first derivative of 



g(T,fc-l) 



with respect to r, yielding 



1 + r) 2fe - 2 + (fc - 1)(1 + r) fe - 2 (l - r) fe - (fc - 1)(1 - r) fe - 2 (l + r) fe + (1 - r) 



\2fc-2 



(B.l) 



Denote the enumerator of (B.l I by /(r). It is easy to see that /(0) = 0. Since the denominator of (B.l 
is always positive, it suffices to show that f(r) < for any r > 0. 
To continue, one can verify that 

/( T ) = _(l +r )2^2 + (1 _ r) 2fc-2 + (fc _ 1)(1 _ r 2 ) fc-2 [(1 _ T) 2_ (1 + r) 2 ] 

^2fc-2 , n ^2fc-2 4 r(A; _l)(l_ r 2 )fc- 2 

_2\fc-2 



-(l + r) 2fe - 2 + (l-r) 

fc-2 



i=0 

r/, '~ 2 '2fc- 2\ „„■ . ^ //.' - 2 N 



-2r 



-2r 



(2k -2\ 2 , /jfe-2\ 
.i=0 v 7 i=0 v 7 



i_2i 



(-l)V 



E 



+ 



o<i<fc-2: even 

2fc-2 
2« + l 



2fc - 2 
2i + l 



+ 2(fc-l) 



k — 2 



,'2i 



E 



0<i<fc-2: Odd 



2(fc-l) 



k — 2 



J2i 



< -2t 



E 



o<i<k-2-. even 



2fc- 2 
2* + l 



+ 2(fc- 1) 



fc-2 



-2/ 



< 



(B.2) 
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for any r > 0. In (B.2), the first inequality is due to the fact that for any odd i < k — 2 



2k -2\ _ /2k -3\ /2k -3 
2i + l) \2i + l) + V 2i 

'/.- - 2\ /k - 1\ (k - 2\ (k - i 



> 2(k-l 



i ) \i + 1 J V * /V i 
k - 2 N 



and for i = — 2 when is odd, 



From ( B.2[ ), Property PI follows. 



Since q,a?"2 > Q>A?~i > L it is easy to see that 



] ■ frA - \ _ - ^A^AH 24aC6A^1 4a c 6A?"2 2t^C 6A ^2 



n ' ' / n n n n 



/ - i?in - r 2 i?2 

0. (B.3) 



On the other hand, one can verify that for any k > 1, 

g(r,k-l) 7r(k) 



which implies that 



->+oo g(r,k) k 



bA \ t {1) t {3) 



l im / I — , 1,t \ = I — ^tt (cftAn) - ^tt (c 6A r 2 ) . (B.4) 
r-s>+oo \ n I n n 



Property P2 now follows from (B.3i, (B.4i, and Property PI. This completes the proof of Lemma [2] 



Lemma 3. For fixed ^ and I, P as a function of £ is strictly decreasing over £ G (0,1/2). 

Proof of Lemma |5j To show that P(^,T,£) is strictly decreasing over £ G (0, f/2), take its first 
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derivative, yielding 



dP 



1 - t/l Zdr 
in := In t — 



42 (1 + r) 



(1-r) 



+ r 2 c b A 



4a (1 + r) 

n 

42 (1 + r) 



2riC bA -i 



(1 - r ) 2 ^ c ^-! 5r 



5(r, 2ric feA ) 
a - 1 - (1 -r) 



0£ 



dr 



n 



+ 2r 2 c bA 



42(l + r) 2 ^- 



- (1 - r) 2 ^^- 1 dr 



#(t, 2r 2 c bA ) 



Note that 



(l + r )fc + (l_ r )* 

(l + r )fc-i( 1 + r ) + (l_ r )fc-i(l_ r ) 

r [(1 + r) fc - x - (1 - r)^ 1 ! + 5 (r, fc - 1) 



and hence 



(1 + t 



,fc-i 



(1-r 



fffcfc) - g{T,k- l) 



Plugging the above equality into ( B.5[ ) yields 



1 - idr 

In — — lnr — 

r a£ 



+ 



+ 



+ 



+ 



HCfeA 4a f 

t n 

(2) 

2ric bA ^a 
r n 
+ (3) 

^20) A ^A 

t n 

(4) 

t n 



g(r,ric b A ~ 1) 
g(r, ric bA ) 

_ ff(r, 2ric fcA ~ 1) 
g(r, 2nc bA ) 

g(r,r 2 c b A ~ 1) 
g(r, r 2 c bA ) 

_ ff(r, 2r 2 c bA ~ 1) 
g(r, 2r 2 c bA ) 



Or 

at 



dr 



dr 

at; 



dr 



In In r 



(B.5) 



(B.6) 



where the second step comes from the fact that r is the solution to (|4.2j) and from the identity (|4.6 



Note that r = 1 is the solution to ( 4.2 1 when £ = | , and therefore by Lemma 2 < r < 1 whenever 



January 26, 2012 



DRAFT 



44 



£ G (0, Z/2). Furthermore, it can be verified that for any r G (0, 1) 

g(r,k-l) (l + r )*-i + (l - r)*" 1 



#0, k ) 

which, coupled with (|4.2|), implies 



> 



1 



(l + r) fc + (l-r) fc ' 1 + t 

f <r-« 



1 + r 



or 



t > 



for £ G (0,1/2). Plugging the above inequality into (B.6), we have 

dP 

for £ G (0, J/2). This completes the proof of Lemma [3] 

Lemma 4. For fixed ^ and Z~ P (^,l,£) > P (^,1,1 - £) for < £ < 1/2. 
Proof of Lemma |4| First, we consider the case where 

tw t( 3 ) r 

vr(c 6A ri) H 7r(c b A^2) < £ < 7, 

n n 2 



Define 



P 



bA 



71 



-m c (£//")-einr 



Ha , g( r » r i c bA) , 2ric bA ) 

n n 2 n n 2 

+ 4a ln g( r » r 2CbA) 4a ln 2r 2 c bA ) 



7i 



ri 



and rg as the solution to 



Then it is easy to observe that 



bA 



11 



,l,T 



P 



bA 



11 



U) P[ b -^,l,tn 



Note that when £ < 1/2, r 5 < 1. For r < 1, 
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and 

'6A 

n 



, Ha ! ff( r Vic fcA ) 4a ln g(r 1 ,2r 1 c bA ) 
n 2 n 2 

4a ln g(T~ 1 »T-2C6A) 4a ln 9(T~ 1 ,2r 2 c bA ) 
n n 2 n n 2 

< -ZH (£/Z) + (/-£) In r 

+ 4a ln ff( r > r iQ>A) 4a ln s( r > 2nc 6A ) 



4a ln 9(t, r 2 c bA ) 4a ln g( r » 2r 2CbA) 

-rH e (e/r)-einr 

, 4a ln 9(r, ric bA ) 4a ln g(r, 2ric feA ) 

n 2 n 2 

4a ln gQj r 2 c bA) 41 1 2r 2 c bA ) 

n n 2 n n 2 



where the third step is due to (4.6). Therefore, 



Now it can be verified that 

EMM = zi±l(viM 

<9r r 

and since [ T, r) is an increasing function of r, it is easy to see that 9P ( g/^' r ^ < for r < T£ and 
q/^' T ^ > for r > T£. Therefore, is the value that minimizes the function P £, t) given 

£. In the other words, 



P[ b Al,^)< P ( b AUr 
* n J \ n 



for any r > 0. In total, we have 



x n J \ n 

\ n ? 

> pf—JJ-tn^ 

\ n q 
\ n 
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Now if 



t W t(3) 

£ < — Tr(cbAri) H 7r(c bA ^2), 

n n 



then P(^,l,l-£) = -oo, and P(^,l,£) >P(^,l,l-£) is obvious. For 

i« t (3) 

£ = vr(c fe A^i) H 7r(c b A^2), 

n n 



it can be shown that 



for r > 0. Then 



ap(^,u"-g,r) 



< 



P 



bA 



n 



ri—,i,z, Te 

n 



> P 



bA 



n 



,1,1 -£,t. 



6A 



-i 



> lim P ( — ,1,1 -£,t 



P 



bA 



n 



-,!,!-£ 



where the last equality is due to (4^41. This completes the proof of Lemma [4] 



Lemma 5. For — < £ < I 



P 



in 

bA 



bA 



l,£) < ln2 + 2£exp 

n I n 



2 1 
I 



(c bA ri - 1) 



bA 

H exp 

n 



2£ 



-ric b A 



Proof of Lemma [5j Let T£ be the solution to the equation 



n 



£ T£ 1 

^ - < rt < 1 or = < — ^— < - 
1 - £ I ~ 4 ~ I ~ 1 + r 5 - 2 



From the proof of Lemma |4| we know that 

whenever £ < ^. Furthermore, it can be verified that 

f(x) - l + ( l ~ x ) k - 1 

is strictly decreasing for | < x < 1, where k is an integer no less than 2. To see this is the case, we 
have 



[1 + (1 - x) k ] (k - 1)(1 - x) k ~ 2 +[! + (!- a:)*" 1 ] k(l 



x 



[l + {l-x) k ] 2 



[l + (l-x 



,fcl 2 



1 — kx + (1 — x) 



< 
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for | < x < 1. Now assume that 



Then 



Therefore, 



Similarly, 



1 + Tg / 



g (r^CbATL - 1) 
g{T^,c bA ri) 



1 !+(!- 



2r s V-Ari-l 

1 + T £ 



1 +r € 



1 + 



< 



C6A7"i — 1 



+ T € 1 



+ 



M) 



< 



g (rg,2c b An - 1) 
9 (7"f , 2c 6 at-i) 



g(^,c bA r 2 - 1) 
9(^,c 6 Ar 2 ) 



g(T^2c bA r 2 - 1) 
g(r^,2c bA r 2 ) 



1 + r € 
1 

1 +r 5 
1 

1 + r € 

< 
< 
< 
< 
< 
< 



1 + 



1 + 



M) 



i + fi-¥ 



fi _ M 

z V i 



I 

CbATl-1 



i + (i-¥ 



i + Hfi-S 



Ci,A»*l 



CbAri-1 



1 +r f 

1 

1 +r f 
1 

1 +r f 
1 

1 +r f 
1 

i 

1 + T£ 



1+ f (1 
1+ f (1 
1+ f (1 



i 



1? 



2c(,Ari— 1 



CbAri-1 



CbAr 2 -l 



CbAri-1 



2C(,Ar 2 -l 
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Thus, 



bA 



n 



4 A C &A»1 g(r, c b ATi - 1) 2t^c 6A ri g(r, 2c bA n - 1) 



n g(T,c bA ri) 



n 



g(r, 2c bA ri) 



4a c ^a^2 g(r, c bA r 2 - 1) 2tflc bA r 2 g(r, 2c bA r 2 - 1) 



> I 



n 
I 



g(r, c bA r 2 ) 



1 +T£ 



1 + 



2 A 
i 



i 



n 

CbA.ri—1 



g(r, 2c bA r 2 ) 



where in the last step, the identity (|4.6|) was applied. This implies that 



On the other hand, 



Again, 



1 + r c 



< 



I 



1 + 



2C 



2C 



g(n,r lCbA ) = (1 + r ? ) riC - + (1 - r { ) 



\r-iCbA 



= (l + r € ) riC - 
< (1 + r € ) riCi)A 



1+1 



2 X r lCtAn 



1+1 



1+T{ 



5(T 5 ,2r lQ , A ) < (l + r ? ) 2riC - 

< (1 + r f ) 2riC6A 
5(r ? ,r 2 c feA ) < (l+r e r ctA 

< (1 + r f ) r2CbA 

g(r^2r 2 c bA ) < (1 + r e ) 2r2C " A 

< (1 + r c ) 2r2Ci)A 



1+1 



l +l l 
1+fl- 



1+ 1 



1+ Y 
i + 6 



tJ 



2 A 
i 

j 
i 

2 A 



2r 2 Ci,A 



Z J 

T 
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By combining the above inequalities with the identities (4.5) and (4.6 1, we have 

= -IB. ((/I) -(lnT C 



n 



+ 4a ln 9(n,r lCbA ) + 4a ln g(rg,2r lCfeA ) 



n 



v 



+ 4a ln 9(n,r 2 c bA ) + 4a ln g(Tg,2r 2 c feA ) 



n 



n 



< -ZH e ($/l) -flnr e + Zln(l + r f ) - — ln2 + — In 



1+1 



2C 



< -ZH e f£/Z) -£ln^— + Zln(l + T f ) - — ln2+ — ln 

v y Z — £ n n 



1+1 



2£ 



< -'H (f /!) - ( In |4t + fT7 + ' ln 



1 + 



CbAfl-1 



6A 6A 
— ln 2 H ln 

n n 



< 



< 



bA 

n 

bA 

n 



ln 2 + 2£ 1 



1+1 



2C 



2 A 
i 



+ 



bA 



n 



2<: \ '' ' '■'■a 

T 



ln 2 + 2£ exp 



2£ 
Z 



(cbA^l - 1) 



bA f 2£ 

H exp — r ric 6A 

n V Z 



This completes the proof of Lemma [5] 



Lemma 6. Given < £ < 2> the following properties hold: 
SI /or 1 < 5 < f - 1, 

A, _ . „/(6+l)A 



In 2 < P 



n 



n 



bA 



l,t]-P[—,l,Z] < Q 



where 



( 



Q 



(6 + l)A 



In 2 - In 



1 + 



n 



2(l-f 



A 



flC(b + l)A 



V 



i + (i-¥ 



2riC (b+ i)A 



S2 for 0<R 1 <R 2 <1, 

-(R 2 -Ri)ln2<P (R 2 , 1, £) - P {Rul £) <-(R 2 - Ri)Qz,r 2 

where 



( 



Q^R 2 = ln 2 - ln 



1 + 



2(1-^ 



riC R2 



V 



i + (i- ¥ 



2riCjj 2 



with cr 2 = 2 r io S2 -^2^ w hi c h shows that P(R, I, £) strictly decreasing with respect to R G (0, 1], 
awe? Lipschitz-Continuous with constant ln2; and 
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S3 ]im RrM >P(Rj,S) = 0. 

Proof of Lemma^ Let P(^,l, £, r) be the function denned in the proof of Lemma [4] Furthermore, 
let and 77,+ 1 be the solution to ( |4.2| ) for 6 and 6 + 1, respectively. From the proof of Lemma |4j it 
follows that given I, £), the function P (^,1, £, T ) achieves its minimum at r = 77,, and hence 

\ n J \ n 

< pf^+r,^ 

V n 

Therefore, 

p(<^,«)-p(^,«) < p(^X^-p( b Au,n 

\ n J \ n J \ n J \ n 

Now we have 

CbA = vi > ^i = C(6+1)A = 2-^ ^ > 2-rio g2 ^+11 = C J±. 

To continue, we distinguish between two cases: (1) q,a = C(b+i)A> an d (2) Q,a = 2c(j, + x)A- I n case (1)' 
i.e., when |~log 2 ^] = [log 2 n ], we have 

\ n J \ n 

r (b+l)A E (b+1)A . 9 { T b, nC(6+i)Aj C (6+1)A r 6A 5 (, r 6> ^ r lC(b + i)Aj 

rT~ ~ ~ 2 rT~ ~ n_ ~~2 

,(3) ,(3) . , (4) (4) . . 

>+i)A _ r bA g (T b ,r 2 C( fc+ i) A j l (fe+i)A _ r bA g (7ft,2r 2 C( 6+ i)Aj 
n n 2 n n 2 



Meanwhile, 



Vi)A . / 2(6 + 1) A 

— — = mm ' 



n 

= min 

> min 

t (1) 
Ha 

n 

Furthermore, it can be verified that 



1 2j6 + l)A _ 2[log2 <^a 1 ^ R ^ log2 ( 5 ±pA 1 1 

|2j6+_i)A _ 2[log2 ^i ;jRi2 rio g2 
|^_ 2 rio g2 ^i^ l2 rio g2 ^i| 



JX) , o/ 2 ) .(1) , ,(2) 

t (frfl)A ^ Zt (b+1)A _ ^ 2riog2 Oh^] _ fl 2r i og2 = %a + 2t bA 

n n 
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Therefore, 



2(t (2) -t {2) ) / 
1 {Ha l (b+l)A) _ V+1)A _ 



(1) 
6A 



> 0. 



n 



Similarly, we have 



2 (t {4) - i (4) 

z ybA Z (b+1)A 



n 



n 



M _ A3) 
Z (b+1)A l bA 

n 



> 0. 



Consequently, 



(6 + l)A 



n 



bA 



n 



f( 2 ) _ +( 2 ) 
fc 6A b (b+l)A 



n 



ln2 + lng 2 (r b ,r lC(b+1)A )\ 
5 (T 6 ,2ric (b+1 ) A j y 



+ 



L bA L (b+1)AJ 



In 2 + In 



n 



At the same time, 



g 2 (T b ,r lC{b+1)A ) 
g (n, 2ric (6+1)A ) 



ff 2 (Tfc,r 2 c (fe+1 ) A ) \ 
g (r 6 ,2r 2 c (fe+1)A ) y ' 

[(1 + r &) riC «'+ 1 > A + (1 - r fe ) riC <"+ 1 ) a ] 2 
(1 _|_ r6 )2riC (i>+1)A + (1 - 7^)2ric (b+ i )A 

2(1 + Tb ) riC «-+v* (1 - r 6 ) riC ( ! '+ 1 ) A 
+ (1 + Tb )2r lC(b+1)A + (1 - T6 )2r 1 c (i , + i)A 



1 + 



2(1- 

1 1 + T b 



1+1 



From the proof of Lemma [5] 

< 1 - 

On the other hand, it is easily verified that 



2t„ 

l+T b 



2r b 2f 

— — < 1- -4 < 1. 

1 + r 6 - Z 



2riC( 6+ i)A 



/(*) 



2a; 



1 + x 2 



is an increasing function for x 6 [0, 1). Therefore, 

,2 



flC(b + l)A 



3 (r fe ,2ric (;)+ i )A ) 



2riC( b +i)A ' 
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Similarly, 



g 2 ("71, r 2 c (6+1 ) A ) < 



2 ( 1 _ 2|y2c (i)+1)A 



ff(r b ,2r 2C(6+1 )A) " 1+ ( 1 _|) 2r2C(b+1,A 



And finally, 



< 1 + 



2 (^l _ H5y iC<b+1)A 

/ ne\ 2r-iC( 6+ i)A ' 

1+ v-v 



< 



n J \ n 

\ z bA C (6+1)A J + \ l bA l (b+l)A J 



11 



(M-l)A 



^ r (6+l)A z bA J \ l bA l (b+l)A J \ z (b+l)A z bA J \ z bA Z (b+1)A 

= 

n ? > 
\^(b+l)A + r (f>+l)A + C (6+1)A + r (6+l)A J ^ r f>A + l bA + z bA ^ z bA) 

= Qf (b + l)A 

n S ' n 

A 

— (6+l)A . 

n S ' n 

Using a similar argument, we can show that 



(b + l)A 



n / \ n J \ n J \ n 

(4a - z ^+i)a) ( ln2 {ln 9 2 (Tb+i,r lC(b+1)A ) \ 
n \ 9 (t&+i,27-ic (&+ i)a) J 

+ (4a ~ *(£-i)a) /_ + ^ g 2 (r6+i,r 2 c (6+1)A ) \ 

\ z bA Z (b+1)A ) + I z bA z {b+l)A) 

> — — In 2 

n 

A 

= In 2. 

n 

This completes the proof of Property SI in case (1). 

In case (2), i.e. when |~log 2 ^] = |~log 2 (fe+ n 1)A ] - 1, we have 

p(^, i -,^)-p(^,«„) 

4&+i)A ln 9 ("7"6,nc( 6+ i)A) 4b+i)A ~ 42 ln g (t&, 2ric (6+1)A ) _ 42 ln g ( r fc> 4r iC(b+i)A) 
n 2 n 2 n 2 
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t( (b+i)A ln g (7b, r 2 c (b+1 ) A ) *(&+i)A ~ ^ g (7b, 2r 2 c (b+ i )A ) _ 4a ln g ( T b, 4r 2 c (fe+1)A ) 
n 2 n 2 n 2 



On the other hand, 



i 



(i) 

(b+l)A 



+ 2t 



(2) 

(b+l)A 



n 



(b+l)A n 



9/ W -4- 4f( 2 ) 



n 



which implies that 



,(2) 



b {b+l)A %A 

n 



_ Ji) 9 J2) _ ,(i) 



"bA "(b+l)A 
n 



/2 



Similarly, 



,(4) _ ,(3) „ (4) _ (3) 
\b+l)A L bA _ zt bA fc (b+l)A 



/2 



n 



n 



and therefore, 



< 



(6 + l)A 



n 



t 



(i) 
(b+l)A 

n 

,(3) 



/2 



9 2 (7b,ric (b+1 ) A ^ 



-ln2 + ln 



+ 



"(b+l)A 



/2 



n 



In 2 + ln 



g (n, 2riC( 6+ i )A ) 
# 2 (7b, r 2 C( 6+ i )A ) 



+ 



£ (2) 
c bA 

n 



- ln 2 + ln 



g (n, 2r 2 c (6+1)A ) 



+ 



^bA 
n 



ln 2 + ln 



g 2 (n, 2riC( 6+ i )A ) 
5 (7b, 4ric (6+1)A ) 

5 2 (r 6 ,2r 2 c (5+ i) A 



5 (r 6 ,4r 2 c (b+ i )A ) 



>+l)A 



l z + l bA + C (b+1)A 



/2 + t 



(4) 
6A 



n 



Q. 



(b + l)A 



A 



n 



Q. 



(b + l)A 



where the last step is due to the fact that 



/2 + t 



(2) , ,(3) 



t!,+ \)A> ~ ' 'bA t~ *(b+l)A/ 2 + 4a 



(1) 

(6+l)A 



( 



n 



I I, \ ' III. I A ^ //, A ~ ! ' 



(1) _ ,(2) 
bA t (6+l)A 



(2) , ,(3) 
bA r (b+l)A 



^(3) _ (4) \ _ (4) 
\ l bA r (b+l)A J l bA 



n 



(t {1) + t (2) + t (3) + t {4) - (t {1) + t {2) + t (3) + t (i) ) 
^(b+^A ^ r (b+l)A + r (b+l)A + r (b+l)A J ^ r bA + r bA + r bA + r bA ^ 



n 



A 

n 
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In a similar manner, we have 



n J \ n 

n I \ n 



t {1) 12 
L (b+1)A' Z 



+ 



n 

*(3) 

C (b+1)A 

n 



/2 



In 2 + ln 



In 2 + In 



g {T b+ i,ric {b+1 ) A ) 
g (r 6+ i,2ric (6+1 ) A ) 

g 2 (r b+ i,r 2 C( 6+ i) A ) 
5 (T6+l, 2 ^2C( fe+ i) A ) 



+ 



£ (2) 
l bA 

n 



+ 



t {4) 
%A 

n 



In 2 + In 



In 2 + In 



g 2 (r b+ i,2ric (6+1 ) A ) 
9 (T6 + i,4ric (6+ i) A ) 

g 2 (^+i,2r 2 c (b+1)A ) 
g (r 6+1 ,4r 2 c (6+1)A ) 



> 



*M / 2 + ^ 2 > + /2 + t (4) 

76+ 1W z ^ L bA ^ L (b+1)A/*^ . 



bA 



In 2 



n 



A 



In 2 . 



n 



The completes the proof of Property SI in case (2). 
Property S2 can be proved in a similar manner. 

Now let us move to the proof of Property S3. By Lemma HI for £ G (0, 1/2], 



which implies that 



P(R,l,0 > P(R,l,l/2) = -i?ln2 



lim P(R,l,0 > . 



At the same time, let tr be the solution to the equation ( |4.2[ ) with — = R, we have 

P(R,l,0 = P(RM,tr) 



< -m e (e/0 - £ In I + /"In f 1 + 4 



where the third step follows the fact that ~ < 1 and yL — jf-^J < yl + j^j for any positive integer 
k. And therefore, 



which further yields 



This completes the proof of Lemma [6] 



lim P(R,l,£) < 
R->0 



lim P(R,l,0 = . 
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Lemma 7. Suppose that I is an odd integer. Then for any given ^ > 0.75, P {^,h£) ' s a strictly 
decreasing function of £ in the range ( ™ , I 



.<i) 



Proof of Lemma 7 Since I is an odd integer, we have R\ = 1, R 2 = 0, and hence t$ = t^) = 0. 
Furthermore, whenever ^ > 0.75 > 0.5, one has q,a = 1, which, coupled with R\ = 1, implies 



£ {1) 
n 



(2bA 



mm ■ 



■n 



and 



.(2) 
l bA 

n 



In view of (B.6l, it suffices to show that 



or equivalently, 



bA t 

n 

T > 
1 

f+T 



(1) 
6A 

n 



2feA 



n 



6A 



?? 



Hi 

i-e/r 



<i-cA 



, where r is the solution to the equation (|4.2|). By Lemma 2 and the fact that r = 1 



when £ = 2 , we have r > 1 for £ G ( i , I — ^ 



4.2|) can be further simplified as 



/26A 



n 



. Moreover, according to the discussion above, equation 
g(r,l-l) , / 26A\ g(r,2l-l) 



9(rJ) 



or 



1 

1+T 



/26A \ '■ + 
V n 



T-l 
T + l 

T — 1 
T+l 



+ 2 



z-i 



n 



g(r,2l) 



+ 2 



26A 

n 



1 



1 + 



T-l 
T + l 

T-l 
T+l 



2Z-1 



2/ 



i-£/L 



Let z = ^rj, and the lemma is proved by showing that 



/26A _ 

\ n 

for z G (0, 1). Towards this, note that 



1 + z 



l-i 



1 - z l 



+ 2 



26A\ 1 -z 2 '" 1 
n / l + z 2Z " 



> 1 



1 + z 



:-l 



1 - z' 



> 1 > 



1 - z 



2i— 1 



1 + 3 



2/ 
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and 



2bA 



11 



l + z 



i-i 



1 - z 



+ 2 



2b A \ l-z 2 '- 1 



2 1 1 - / + l + z 21 j + 



n J l + z 21 
2bA 3\ / l + z 1 ' 1 



n 



1 - z 



21-1 



1-Z 



l + z 



21 



1 (l + z 1 ' 1 1-z 

> - — =- H = 

" 2 I 1 - z l l + z u 



21-1 



when — > 0.75. Furthermore, 



l + z 



l-l 



l-z L 



+ 



1 - z 



21-1 



l + Z 



21 



> 2\ 



1 + z 1 - 1 1 - z 21 - 1 
1-z 1 1 + Z 2T 



l + z 1 - 1 1-z 



21-1 



l + z 



21 



1 - Z l 



> 2 



since < z < 1. This completes the proof of Lemma [7] 

Appendix C 
Proof of Theorem [1] 

Given x n and y n , let j = j(x n ,y n ) be the number of interactions at the time the decoder sends bit 1 



to the encoder. From <\3.l\ and (|3.2|), it follows that 

^+H( e ) + £ ifj<A/n 



r f (x n ,y n \l n ) 



1 + Vn + H(e) + ^ otherwise 



and 



r fe (x n )2 /"|X n ) 



n 



(CI) 



(C2) 



Since A ~ y/n and j < ^ + 1 according to Algorithm [TJ ( |4.8| ) follows immediately. 

In view of the description of Algorithm [TJ it is not hard to see that at the (j — l)th interaction, one 
always has 

iy_i < h n {x n \y n ) . (C.3) 
We now distinguish between two cases: (1) h n {x n \y n ) < Fit, and (2) h n (x n \y n ) > Til. In case (1), it 



follows from (C.3 1 that 



< 



n 
A 



(C4) 



and 



1 

m~2 



n ' ' / A 2 2n 4 



A 



n 



< K(x n \y n ) 
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or equivalently 



h n (x n \y n ) + 



A 



n 



3R1 nl 1 , nl 

ln2 + -V-ln 1 In — 

A 2 2n 4 



By Lemma |6j P (R,l,l\e) is strictly decreasing with respect to R. Therefore, 

tt <fl w )(£i/ln(lV)) . 



(C.5) 



Combining (C.ll, (C.4i, and (C.5 1 together yields 



r f (x n ,y n pn) < R { L % (e,h n (x n \y n ))+R(e) + 



2A 

n 

This completes the proof of ( |4.7[ ) in case (1). 

In case (2), j could be strictly greater than ^. Regardless of the value of j, in case (2), one always 
has 

A 



r f (x n ,y n \l n ) < l + r ?n + H(e) + 



= R[f ) (e,h n (x n \y n ))+R(e)+ 2 ^. 

This completes the proof of ( |4.7| ) in case (2). 

Towards bounding the error probability, for any x n G B n and < e < 0.5, define 

B(e, x n ) = L n eB n : -wt(z n - x n ) < e or -wt(z n - x n ) > 1 - el . 

[ n n J 

To proceed, 

P e {l n \x n ,y n } = Pr{xVx"} 

= Pr {x n G B(e, x n )} Pr {x n ± x n \x n G B(e, x n ) } 

+ Pr{f n £ B(e,x n )}Pr{x n ^ x n \x n <£ B(e,x n )} 
< Pr {x n ^ x n |x n G £(e, x n ) } + Pr {x n B(e, x n )} . 
We first consider Pr{£ n ^ i?(e, x™)}. By the union bound, 
Pr{x n i B(e,x n )} 

< Pr {3z n i B(e, x n ) : H^^" = H^ n x n , h n (z n \y n ) < T b for some b, 1 < 6 < J j 
+ Pr {3z ^ £ ) : H nX n^ = H nX nX , H rinnxn z = H^ nxn x } 



n "I 



< ^Pr{3z" ^ J B(e,x n ) : H 



6=1 



iiAxn' 



+ Pr |3z n ^ B(e,x n ) : H n x n z n — H n xnX n , H r/nnxn z n — H^ nxn x n } . 
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Now by Lemma [T| for 1 < b < ^, 
Pr (h 



(6A) n _ tt(6A) n 
z — - n bAxn X 



6Axn x 



} 

Pr{H<$ n (*»-z») = 0*} 



cx-p { « P (~, i, i) + ^1 ln(ne) + i InnC (l - f ) + O(l) 



< exp < n 



n 



3 HI nZ 1 raZ 

L In 1 In — 

A 2 2n 4 



O(l) 



while 



Pv {H nxn Z — Hjjxn^ } 

= Pr{U nxn (z n -x n ) =0 n } 

< exp jnP (1, 1, e) + 3|?| ln(n|) + ^ ln< M - | ) + 0(1) 



< exp < n 



„ , , , , 3R1 , nZ 1 , nl 



n 



+ 0(1) , 



where £ = Z 3 *^" x ") and £ = max { ^, min {£, Z — ^} }. Simple calculation reveals that he < (, < I — he 
for z n £ B(e,x n ), which, together with Lemmas [5] and [4j further implies that 



Pr/H (bA) z n -H {6A) x n \ 



< exp < n 



r> . bA - \ 3 Z , n! 1 . nl 

P — , Z, he) + - J — - In 1 In — 

n A 2 2n 4 



+ 0(1)} = 2 - nr >- A+ °( 1 > 



and 



Pr{H nxn 2i — H nxn x n y 
< exp | n 

Now by the union bound again, for 1 < b < ^, 



„ /, 7 , n. 3TZ1 , nl 1 . nZ 
^(M,/i e ) + ^ln y + -ln T 



+ 0(1)} = 2~ n ( 1 -'?")- A+0 ( 1 ). 



Pr{3z" £ fl( e> x») : H^z" = U^ n x n ,h n (z n \y n ) < F b } 



< \z n i B(e, x n ) : h n {z n \y n ) < T b \ 2 -™ r >- A +°( 1 ) 

< \z n : h n {z n \y n ) < T b \ 2 -™ r "- A + ( 1 ). 

At this point, we invoke the following lemma, which is from (2J: 

Lemma 8. For any y n G y n and any < a < 1, 

|z n : /i„(z n |y n ) < a\ < 2 na 
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where h n {-\-) is the code length function of any decodable code. 
Therefore, we have 

Pr {3z n i B(e,x n ) : H%g n z n = U^ n x n , h n (z n \y n ) < T b ) < 2^°^. 
At the same time, 

PrjEb™ B(e,x n ) : ~H. nX nZ n = H n xnX n ,H r/nnxn z n = H r? ^ )1><n x n } 

< Yl Pr {Knxn(z n ~ x n ) = n } Pr {H' Vnnxn (z n - x n ) = 0""™} 

< 2 -A+0(l)_ 

To sum up, we have shown that 

Pr{x n i B{e,x n )} < 2 - A +Mi+i)+0(i). 

Before moving to the next target Pr{x n / x n \x n G i?(e,x n )}, it is not hard to verify the following 
bound on \B{e, x n )\: 

\B(e,xn\ = 

< 22 nIi ( iz r L ) < 2 nH ( £ ) +1 
Now suppose x n G B(e,x n ), then x ra G B(e,x n ), which, according to Algorithm [TJ implies that 

Pr {x" * x« |x« G 5(6, x") } = Pr {3z« G 5(6, *")/{*"} : H^ H(e)+A)xn ^ = H;; H(e)+A)xn x"} 

< \B{e,x n )\2- nH ^ +A 

< 2 -A+G(l)_ 

In summary, 

P e {X n |x n ,y n } < Pr{x"/x"|x^G J B(e,x n )}+Pr{x n G' J B(e,x n )} 

< 2 -A+0(l) + 2 -A+log 2 (f +l)+0(l) 

< 2 -A+log 2 (f+l)+0(l)_ 

The theorem is proved. 
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Appendix D 
Doubly Asymptotical Performance 



In the appendix, we prove Propositions [T] and [2] and Theorem [2] 

Proof of Proposition [IJ In view of Lemma |6j it follows from the definition of (e, h) that 

Rl(z) ( e > h) is the solution to 

-P (R,l,he) = /iln2 



if Mn2 < -P (1,1, he), and 



R L(g) (e,h) = 2 + —P(l,l,he) 



otherwise. On the other hand, in view of the fact that l\e > and of Lemma jij for R G (0, 1], 



P(R,l,he) < -i?ln2 + 2/ieexp =- (c R n - 1) 

< -i?ln2 + 2/ieexp - l) 

where c;? =2-^2^1 > i. Now if Mn2 > -P (l,l,he), then 



+ iZexp I p r i c « 



r L( 2 ) (e, /i) = Rl(z) (e, M + H(e) - /i 

< 2+ i ^ 2 P(l,/",/ 1 e)+H(e) 



. 4/ l£ 
^ IrT2 eXP 



2Zie 



(W-i) 



If ft In 2 < -P(l,M x e), then 

ft In 2 = -P(R L(z) (e,h),l,he) 

> Rl(z) ( e ) ^) m 2 — 2lieexp 

which implies that 



2/ie 



(W-i) 



(e, h) <h + ^ exp 



In 2 



2jl€ 



(w-i) 



1 / 2Zie. r . 

+ hT2 exp 



Therefore, 



rL(z)(e,h) 



R L{z) (e, ft) + H(e 
2Zie 



2/ie 
* TrT2 eXP 



ft 

(W-i) 



+ hT2 exp 



2/ie. 



W + H(6) 



(D.l) 



(D.2) 



Combining ( D.l I with ( D.l) completes the proof of Proposition [T] 
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Proof of Proposition |2| Note that k > e'i , which implies that 

, , In k , , , kl 
kh — = hi nk>2> w 

and therefore, we can apply Proposition 1 on ru z h\ (^r, h), resulting in 



r L(z k ) 



In k 

~k 



Ah In A; 
In 2 



exp 



2h {[kl\ - 1) 



It is easily verified that 



On the other hand, 



H 



In k 

~k 



In k 



O 



+ fn2 eXP 



In 2 k 



2h[kl\ 
kl 



Ink) + H 



In k 

IT 



2h[kl\ > 2h([kl\-l) > 4 (LfeZJ - 1) > i 



A;/ 



Therefore, 



In k 



,h) =o 



In k 
~k~ 



kl 



+ o 



+ o 



kl 



In k 



O 



In 2 k 



Proof of Theorem |2} In view of Theorem [T] ( |4. 14 ) and ( 4.15 1 follow immediately. Thus it suffices to 
prove (413 1. From Theorem [T] again, we have 



In k 

~2k 



+ 



2A 



n 



(D.3) 



Let 5 > be a small number to be specified later. In view of the definition of Rj^rL (e> h n {x n \y n )) and 
Lemma J^J it is not hard to verify that R^l^ (t,h n (x n \y n )) is non-decreasing as h n (x n \y n ) increases. 
This, coupled with ( D.3 1 and ( |4.12| ), implies that with probability one 



rf(x^\X n ^L( Z % l ^<R^ k) (~H(X|Y) + *) + H 
for sufficiently large n. Applying Propositions [1] and [2] to ( |D4| ), we have 

limsupr/^n^f^),^)) < H(.X\Y) + 5 + r L{zk) 

= B(X\Y) + 5 + 



In k 

~2k 



+ 



2A 



7i 



(D.4) 



In k 

~2k' 



r(x\y) + 5 



In 2 k 



(D.5) 



with probability one. Letting 8 — > and then /c — > oo in ( |D.5| ) yields 



lim sup lim sup r f I X n ,Y Tl 



In A; 

~2~aT 



< H(x|y) 



with probability one. This, coupled with the converse [2, Theorem 3], implies ( |4.13| ). This competes the 
proof of Theorem [2] 
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Appendix E 
Proof of Theorem[3] 

In view of Theorem [TJ it suffices to prove (5^2) and (5^1. Note that from the proof of Theorem [T] 
and the description of Algorithm [3] it can be seen that for any sequence of source-side information pairs 

(X n ,Y n ), 

A t \ R { u z) (e, H {^wt(X n - Y n ) + lr ^ ±i )) if wt(X n - Y n ) < 0.5n 



r s \X\Y n \X n ) < - + 
Therefore, 

r/(X„) < -+Pr<! \wt{X n -Y n ) < p + 



otherwise. 



■n 



x E 



n 



Inn 



n 



R 



(A) 
L(*) 



e,H( -wt(X n -Y n ) ) + 



Inn + 1 



± wt (X n -Y n )< P0 + J— 
n V n 



x E 



(A) 
L( 2 ) 



e,H l-wt{X n -Y n ) + 



Inn + 1 



n 



In 7? I 

Po + M — < ~wt {X n - Y n ) < 0.5 



n n 



n 



+ Pr <j -wt (X n - Y n ) > 0.5 \ R§\ (e, 1) 



< 



A 



+ E 



■n 



R 



(A) 
L(z) 



e,H( -wt(X n -Y n ) ) + 



Inn + 1 



^((r-y") <p + ;ln " 



+ Pr J hot (X n - Y n ) > p + y ^ 



where we assume that 



Inn 

n 



Po < 0.5 - 

which always holds for sufficiently large n as po < 0.5. On one hand, given 



-wt (X n -Y n )< Po + J < 0.5 

n V n 



we have 



H -wt(X n -Y n )\ < H U + 



Inn 



n 



< H(p ) + log 2 



1 — po\ /Inn 
Po / V n 
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which further implies that 



E 



Inn + 1 



n I n 

Inn + 1 



1 . wt (X n -Y n )<p + /h '" 



n 



n 



+ log 2 



1 -Po \ Ann 
Po J V n 



On the other hand, by Hoeffding's inequality, 



PrJ h wt {X n -Y n )>p Q + 



< n 



from which ( |5.2| ) is proved. 

Towards showing ( |5.4[ ), we have 
1 



E 
E 



-wt(X n - X n " 



ii 



E 



1 



wt(X n - X ns 



n 



x n yr, 



E 

(x n ,y"-): — wt(x n — y™)<0.5 



Pr{X" = x n ,y n = y n }E 



1 



n 



wt(X n - x n ) 



x n ,y n 



+ ^ Pr{X™ = x n ,Y n = y n }E 

(x n ,y n ): — wt(x"— y")>0.5 



1 



x n ,y n 



< 



Pr{X n = x n ,Y n = y n }E 

(x n ,y n ): — wt(x rl —y n )<0.5 



+ Pr <| ~wt{X n - Y n ) > 0.5 } . 



1 



71 



x\y n 



By Hoeffding's inequality, 



Pr | ^wt(X n - Y n ) > 0.5 j < e -M0-5-p o y 



(E.l) 



(E.2) 



On the other hand, 
1 



E 



n 



x\y n 



1 



Pr <^ -wt(X n - x n ) < e 
n 



x n ,y n }E 



n 



-wt(X n - x n ) 



1 



ii 



wt{X n -x n ) < e,x n ,y r ' 



+ Pr <^ -wt(X n -x n )>e 



n 



x n ,y n > E 



n 



-wt(X n - x n ) 



1 



n 



wt{X n -x n ) > e,x n ,y r 



< e + Pr{ 1 wt(X n -x n )>e 
n 



x n ,y n 



(E.3) 
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Now we would like to bound 



Pr <^ -wt(X n -x n )>e 



n 



x n ,y n 



when ^wt(x n — y n ) < 0.5. By the argument in the proof of Theorem [TJ 



Pr <^ -wt(X n -x n )>e 



x n ,y n 



< Pr |3x n , ^wt(x n - x n ) > e,U ( ^ n (x n - x n ) = bA , j(x n , y n ) < T b for some b, 1 < b < f | 
+ Pr !^x n , ^wt(x n - x n ) > e,U nxn (x n - x n ) = 0", H' Vnnxn {x n - x n ) = 0"" n 

6=1 

+ V Pr <^ 3x n , -wtU 
/ — ' n 

u_\ 0.75™ 1,1 



x n -x n ) = bA , 7 (x n ,y n )<T l 



x n - x n ) > e, H<*f > (x n - x n ) = bA ^(x n ,y n ) < T 



+ Pr jaf", Kvt(x n - x n ) > e,H nxn (x n ~ x n ) = 0", H^ nxn (£ n - x n ) = 0"» n j . 
For 1 < b < L^J, ^ < 0.75 and therefore, 



l{x n : y n )<T b < — <0.75 
n 



which, together with ( |5.1[ ), further implies that 

^t(x n -y n ) < H- x (0.75) 

and 



-wt(x n -x n ) < -wt{x n - y n ) + -wt(x n - y n ) 
n n n 

< 0.5 + IT 1 (0.75) 

< 1- e 

since e < 0.5 - H -1 (0.75). Consequently, we have for any 1 < b < L^^J 

Pr !^x n , ^wt(x n - x n ) > ^H^Cx" - x n ) = feA , 7 (x n ,y") < T b 

= Pr !^x n , e < ^wt(x n -x n )<l-e, H^Jx" - x n ) = bA , 7 (x n , y n ) < T b 
< 2 - A +°( 1 ) 



(E.4) 



(E.5) 
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where the inequality above has been proved in Appendix [c] For b > [ 0, ^ n j + 1, by Lemmas [3] and 



,(i) 



. In view of this, it can be 



6) 



|t| P C) is a strictly decreasing function of £ in the range ^0,/ 

shown by the same technique as in Appendix |c| that for any b > [ °' 7 ^ n \ + 1 

Pr S^x n , Kvt{x n - x n ) > e,H^ n (x n - x n ) = bA , 7 (x",y n ) < T 6 j < 2- A +°« (E. 

and 

Pv^3x n ,^wt(x n -x n ) > e,U nxn (x n - x n ) = n ,H' V7inxn {x n - x n ) = 0^ < 2- A+ °^. (E.7) 
Plugging ( |E3] l, ( |R6l ), and ( |E77| ) into ( gig) yields 



Pr i -wt(X n -x n )>e 
' n 



x n ,y n > < 2~ A+log 2(i +1 ) + °( 1 ) 



(E.8) 



for any (x n ,y n ) with ^wt(x n — y n ) < 0.5. This, combined with ( |E.3j ), ( |E.2| ), and ( |E.1[ ), implies 

< e + 2- A+lo ^(i +1 ) + °( 1 ) + e -2«(o.5- P „) 2 
which completes the proof of ( |5.4| ) and hence of Theorem [3] 



Appendix F 
Proof of TheoremH] 

Note that \5.2\ applies to any value of I, since its proof in Appendix [e] does not rely on the condition 
that I be an odd integer. Then by using Proposition [3] and following the same approach as that in the 
proof of Theorem [2] ( |5.5[ ) is proved, while ( |5.6| ) is obvious. 

What remains is to prove \5.1\ . To this end, let e = Then po < as > ^ 2 (i-2p n ) ) ' ^ e 
same argument as in Appendix [Ej 

1 



n(x n ) 



E 



-wt(X n - X n " 



n 



< Pr{X n = x n ,Y n = y n }E 

+ Pr ( -wt(X n - Y n ) > ^—^ 

In 2 



n 



-wt(X n - x n ) 



x n ,y n 



and 



E 



1 



n 



wt(X n - x n ) 



x n ,y n 



< e + Pr i -wt(X n -x n )>e 
n 



x n ,y n 



given ^wt(x n — y n ) < At the same time, by the decoding procedure of Algorithm [3 

7(X n ,y n )< 7 (X n ,y n ) 
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and therefore 



which further implies that 



-wt(X n - y 11 ) < -wt(x n - y n ) 
n n 



-wt(X n - x n ) < -wt(X n - y n ) + -wt{x n -y n )<l-e. 
n n n 

Consequently, for any (x n ,y n ) with -wt(x n — y n ) < — 



Pr i -wt(X n -x n )>e 
n 



2 ' 

./•" . //" } - Pr { e < \wt{X n -x n )<l-e 



x n ,y n 



< 



2 -A+log 2 (f+l)+0(l) 



where the last inequality has been proved in Appendix [C] The inequality ( |5.7| ) now follows from the fact 
that 

Pr^wt(X n -Y n ) > i-^j < e -M^^o) 2 =e -2n(o.5-^- Pn )^ 
This completes the proof of Theorem |4] 

References 

[1] E.-H. Yang and D.-K. He, "On interactive encoding and decoding for lossless source coding with decoder only side 

information," in Proc. of ISIT'08, July 2008, pp. 419-423. 
[2] , "Interactive encoding and decoding for one way learning: Near lossless recovery with side information at the decoder," 

IEEE Trans. Inf. Theory, vol. 56, no. 4, pp. 1808-1824, 2010. 
[3] E.-H. Yang, A. Kaltchenko, and J. C. Kieffer, "Universal lossless data compression with side information by using a 

conditional mpm grammar transform," IEEE Trans. Inform. Theory, vol. 47, pp. 2130-2150, 2001. 
[4] J. Meng, E.-H. Yang, and D.-K. He, "Linear interactive encoding and decoding for lossless source coding with decoder 

only side information," IEEE Trans. Inf. Theory, vol. 57, no. 8, pp. 5281-5297, Aug. 2011. 
[5] M. Sartipi and F. Fekri, "Distributed source coding in wireless sensor networks using ldpc coding: The entire slepian-wolf 

rate region," in Proc. Wireless Communications and Networking Conference, 2005. 
[6] D. Schonberg, K. Ramchandran, and S. S. Pradhan, "Distributed code constructions for the entire slepian-wolf rate region 

for arbitarily correlated sources," in Proc. IEEE Data Compression Conference, 2004. 
[7] , "Ldpc codes can approach the slepian-wolf bound for general binary sources," in Proc. of fortieth Annual Allerton 

Conference, Urbana-Champaign, IL, Oct. 2002. 
[8] A. D. Liveris, Z. Xiong, and C. N. Georghiades, "Compression of binary sources with side information at the decoder 

using ldpc codes," IEEE Comm. Letters, vol. 6, pp. 440-442, Oct. 2002. 
[9] J. Jiang, D. He, and A. Jagmohan, "Rateless slepian-wolf coding based on rate adaptive low-density-parity-check codes," 

in Proc. of ISIT'07, 2007, pp. 1316 -1320. 
[10] A. W. Eckford and W. Yu, "Rateless slepian-wolf codes," in Proc. of Asilomar Conf. on Signals, Syst., Comput'05, 2005. 
[11] D. Varodayan, A. Aaron, and B. Girod, "Rate-adaptive distributed source coding using low-denstiy-parity-check codes," 

in Thirty-Ninth Asilomar Conference on Signals, Systems and Computers, Oct. 2005, pp. 1203-1207. 



January 26, 2012 



DRAFT 



67 



[12] R. M. Tanner, "A recursive approach to low complexity codes," IEEE Trans. Inform. Theory, vol. 27, pp. 533-547, 1981. 
[13] T. Richardson and R. Urbanke, Modern Coding Theory. Cambridge University Press, 2008. 

[14] J. Ziv and A. Lempel, "A universal algorithm for sequential data compression," IEEE Trans. Inf. Theory, vol. IT-23, no. 
No. 3, pp. 337-343, May 1977. 

[15] , "Compression of lndiwdual sequences via variable-rate coding," IEEE Trans. Inf. Theory, vol. IT-24, no. 5, pp. 

530-536, Sep. 1978. 

[16] J.-C. Kieffer and E.-H. Yang, "Grammar based codes: A new class of universal lossless source codes," IEEE Trans. Inf. 

Theory, vol. IT-46, no. 3, pp. 737-754, May 2000. 
[17] E.-H. Yang and J.-C. Kieffer, "Effcient universal lossless compression algorithms based on a greedy sequential grammar 

transform-part one: Without context models," IEEE Trans. Inf. Theory, vol. IT-46, no. 3, pp. 755-777, May 2000. 
[18] F. R. Kschischang, B. J. Frey, and H. A. Leoliger, "Factor graphs and the sum-product algorithm," IEEE Trans. Inf. Theory, 

vol. IT-47, pp. 498-519, Feb. 2001. 
[19] A. Amraouli, "Lthc: Ldpcopt," online available at the website: http://lthcwww.epfl.ch/research/ldpcopt. 
[20] S. Litsyn and V. Shevelev, "On ensembles of low-density parity-check codes: Asymptotic distance distributions," IEEE 

Trans. Inf. Theory, vol. 48, no. 4, pp. 887-908, April 2002. 
[21] , "Distance distributions in ensembles of irregular low-density parity-check codes," IEEE Trans. Inf. Theory, vol. 49, 

no. 12, pp. 3140-3159, Dec. 2003. 
[22] C. Di, T. J. Richardson, and R. L. Urbanke, "Weight distribution of low-density parity-check codes," IEEE Trans. Inf. 

Theory, vol. 52, no. 11, pp. 4839-4855, Nov. 2006. 
[23] G. Miller and D. Burshtein, "Asymptotical enumeration method for analyzing ldpc codes," IEEE Trans. Inf. Theory, vol. 50, 

no. 6, pp. 1115-1131, June 2004. 
[24] M. R Mineev and A. I. Pavlov, "On the number of (0,l)-matrices with prescribed sums of rows and columns," Doc. Akad. 

Nauk SSSR, vol. 230, pp. 1276-1282, 1976. 
[25] B. McKay, "Asymptotics for 0-1 matrices with prescribed line sums," Enumeration and Design, pp. 225-238, 1984. 
[26] I. Csiszar and J. Korner, Information Theory: Coding Theorems for Discrete Memoryless Systems. Academic Press, INC, 

1981. 



January 26, 2012 



DRAFT 



